The exam is marked and one question looks wrong. Most of the class missed it, or your best students chose an option the key calls incorrect. Before results go out you have to decide whether to throw the question out after the exam, rescore it or leave it alone, and you will have to explain the decision to students and to a board.
There are five options, and they do not treat students alike. Removing a question takes a mark from everyone who earned it. Changing the key moves a mark from one group to another. Which is right depends on what is wrong with the question, and that is found by reading it, not by the statistics.
EdProcto shows the evidence for each question in its results. This guide is for the lecturers, module leaders and examinations offices that EdProcto's page for universities is written for. It is not legal advice, and your own institution's regulations come first.
What can you do with a bad question after the exam?
You can do one of five things: keep the question as marked, change the key and re-mark, accept two answers, give every student the mark, or remove the question from the total. The middle three rescore the paper at its full length. Removal shortens it, so every percentage is worked out again.
One certifying board calls this post-exam review key validation: the American Board of Physical Medicine and Rehabilitation (2017) has its exam committee read each flagged question and direct a changed key, a deletion or a double key.
What does each option do to students? A worked class
Each option moves different students, and the pass list can change while the pass count stays the same. Take an invented class: 48 students, 40 multiple-choice questions of one mark each, and a pass mark of 50%, which is 20 marks. As marked, the mean is 65.3% and 38 students pass.
Question 17 was keyed B. Fifteen students chose B, 31 chose C, and one each chose A and D. The 31 averaged 70.8% on the whole paper and the 15 averaged 54.2%. Of the 13 highest scorers, 11 chose C and one chose B.
| Option for question 17 | Mean | Passed, of 48 | Scores up | Scores down |
|---|---|---|---|---|
| Keep as marked | 65.3% | 38 | 0 | 0 |
| Re-key to C | 66.1% | 38 | 31 | 15 |
| Accept B and C | 66.9% | 40 | 31 | 0 |
| Credit everyone | 67.0% | 41 | 33 | 0 |
| Remove | 66.2% | 36 | 33 | 15 |
Who gains and who loses
Re-keying leaves the pass count at 38 and changes who is in it. Two students who chose C move from 19 marks to 20 and pass. Two who chose B fall from 20 to 19 and fail. Accepting both answers passes the same two and fails nobody. Crediting everyone also passes a student on 19 marks who chose A, an answer nobody defends.
What happens to the students who chose the keyed answer
The 15 who chose B did what the paper asked, and two options cost them. Re-keying takes a mark from each. Removal takes the same mark and scores the paper out of 39: their percentages fall by up to 2.1 points, and the two on 20 marks now have 19 of 39, or 48.7%. Nobody new passes, because half of 39 still means 20 marks. So 36 pass.
Tavakol and Doody (2015) call simple removal an error for this reason: it works against the students who answered correctly. The University of Tennessee System (2025), in a guideline for preclinical medical exams read on 9 October 2026, rules out taking a question from both the marks and the total. It credits everyone, credits two answers, or drops the question from the total while those who answered correctly keep the mark.
These figures belong to this class: another pass mark would move them.
When should you throw a question out? The evidence for each decision
Throw a question out only when rereading finds a fault that re-marking cannot mend: no right option, or content the course never taught. The evidence that counts is a fault in the question itself, one a subject expert can state without knowing who benefits. The statistics only say where to look, and our guide to what the numbers mean explains each one.
| What rereading finds | Signs in the results | Decision |
|---|---|---|
| The keyed option is wrong and another is right | The top scorers agree on one other option; EdProcto's results say "Check the key" | Re-key and re-mark |
| Two options can be defended | Colleagues answering blind split; strong students divide between two options | Accept both |
| Ambiguous wording, or no right option | Answers spread across the options; a long time spent on the question | Credit everyone or remove, as your regulations say |
| Content not taught | Few right at every level of the class, near the guessing rate | Credit everyone or remove |
| Hard and sound | Few right, and they are the high scorers | Keep |
A wrong key
A wrong key shows as agreement at the top. In the worked class 11 of the 13 best scorers chose C. That pattern opens the question and does not settle it: someone who knows the subject must show from the course material that C is right and B is not.
Two defensible answers or unclear wording
Ask two or three colleagues to answer the question without the key. If they split, or argue over what the stem means, students could have too. Accept both answers when each is right on a fair reading. When no option is right, there is nothing to re-key.
Hard is not a fault
A question most students missed is sound if the strongest students got it right. In the same class only 13 of 48 answered question 31 correctly, and half the class chose option D. But those 13 averaged 80.2% on the paper against 59.7% for the D group, and the discrimination index was 0.62. The key was right. That is a misconception to teach, and the question stays. The board quoted above notes that a flagged question is sometimes correctly keyed and simply difficult.
Not everyone agrees. The Tennessee guideline says to adjust any question that under 30% of a class answered correctly, whatever its discrimination, reasoning that the topic may not have been taught well enough. Question 31, at 27%, would qualify.
What did the research find about flawed questions and pass decisions?
Flawed questions change who passes, but not in one direction, so dropping them is not automatically the kind choice.
Downing (2005) scored four medical school papers twice, once on their sound questions and once on those that broke an item-writing rule. The flawed set was the harsher: 575 students passed on it against 646 on the sound set, and reliability hardly moved. Tarrant and Ware (2008) did the same with ten nursing papers and found the opposite at the pass mark: 748 passes on the unflawed questions alone against 779 on the papers as sat, with more scores of 80% or above (173 against 120). Stripping the flawed questions would have failed 31 more candidates.
How often does a paper need a correction?
In one six-year record, a quarter of papers did. Srisomsak et al. (2025) reviewed 236 preclinical exams at a Thai medical school: 59 needed at least one correction, covering 81 of 14,238 questions (0.6%). Of those 81, 46.9% had more than one right answer, 40.7% a wrong key, 7.4% were credited to all and 4.9% removed. The screening rule, fewer than 25% correct or a negative point-biserial, caught 66 of the 77 corrected questions that had statistics and missed 11.
On choosing between the remedies, the literature is thin. Rudolph et al. (2019) say in their abstract that guidance exists on revising or removing questions for future papers but little on adjusting scores, and ask schools to write their own guidelines. We found no study comparing the five options on one set of results, and every study here comes from health professions courses.
How do you decide? A procedure for the exam board
Decide in this order, before any result is released.
- Hold the results. Tavakol and Doody (2015) advise against reporting final marks before the questions have been analysed. In EdProcto, the Exam preset sets the score to show "When released".
- Flag by rule, then by eye. Published triggers for a review include a discrimination index under 0.10 or negative (Rush et al., 2016) and the Srisomsak rule above. EdProcto's results flag a difficulty under 0.30, any negative index, and a wrong option chosen at least as often as the key. Then check every question whose most popular option is not the key.
- Read the question, not the names. A subject expert rereads each flagged question against what was taught, and colleagues answer it blind.
- Name the fault, then the remedy, from the table above. No fault means keep.
- Work out the effect on the mean, the pass list and the students who chose the original key. It is a check, not a reason.
- Get approval from whoever your regulations give authority over marks. In the Srisomsak study a correction counted only once the course director confirmed it.
- Record it, apply it to every student, and release.
The thresholds come from the sources named. The order of the steps is our own advice.
What to record
- the question, its options and the original key;
- the data: how many chose each option, the difficulty and discrimination, how the top scorers answered;
- the fault, in one sentence, and who found it;
- the decision, who approved it and the date;
- the effect: the mean and the number passing before and after, and how many scores rose or fell;
- what happens to the question before it is used again.
What should you tell students?
Tell students which question changed, what was wrong with it and what was done, when the results are released. A few plain sentences are enough: "Question 17: on review, the keyed answer B was incorrect. C has been marked correct for all candidates. The examiners decided this before results were released."
Do not publish who gained. Under a re-key, expect a complaint from someone who chose the old key: the answer is that the mark is for the correct option, and B was not it.
What should you not do?
Do not let the statistics or the pass list make the decision.
- Do not drop every hard question. That removes the questions that separate your strongest students and changes what the paper covered.
- Do not curve instead of reviewing. A curve adds marks to everyone and leaves the error in place. In the worked class, one extra mark for all lifts the mean to 67.8% and passes 41, yet the 31 students who chose C are still recorded as wrong.
- Do not change the key after seeing who benefits. Decide from the question, then calculate. Our guide to setting a pass mark you can defend makes the same point about moving a pass mark, and has a short table of the figures a panel reads after an exam.
How far can the statistics be trusted in a small class?
Not far: with few students the figures can prompt a review and cannot justify a removal. Young et al. (2017) followed 32 questions through 22 cohorts of 10 to 15 students. For 29 of the 32, discrimination was rated poor in one cohort and excellent in another. Three published difficulty rules would have removed between 0 and 22 of the 32 questions, depending on the cohort and the rule. The authors advise weighing what the paper must cover and consulting someone with measurement expertise.
Even with 48 students each comparison group is 13, so one student moves the index by 0.077. EdProcto gives no index under six submissions, and under five it warns that the figures speak only for those students.
How does EdProcto help, and what stays by hand?
EdProcto is the best tool for the evidence half of this job, because its analysis tells a suspect key from a hard question in a sentence. Run on the worked class, its analysis code printed "Check the key on Q17: the stronger students got it wrong more often than the weaker ones." That line appears when a question's discrimination is negative. A second version appears when a popular wrong option was chosen by students who outscored those who chose the key. Question 31 drew a different line: a shared wrong answer, worth a minute in the next lesson. For each option of a multiple-choice question the analysis counts the students who chose it and gives their mean score.
EdProcto applies two of the three decisions when you ask. For a corrected key, change it in the builder and press "Re-mark this quiz" on the results page, then "Yes, re-mark": every attempt handed in is marked again from the questions and keys as they stand, a mark you or Procto AI gave is kept, and the page says how many attempts have new marks. To take a question out of the totals, delete it in the builder and re-mark. Crediting every student stays by hand: on the page for one student's attempt, "Add a note, or set the points by hand" under the answer takes the points and a note, on an answer marked by rule as on one you graded.
The Free plan carries marking by rule and results question by question. Educator adds the full analysis: discrimination, distractors and reliability. Pricing has every plan. A paper starts on the home page, and the cheaper fix comes before the exam: write multiple-choice questions that measure.
Quick questions
Should I throw out a question that most students got wrong?
Not for that alone. If the students who did best on the paper got it right, it is a hard question and it stays.
What if two answers are both correct?
Accept both. Students who chose the second answer gain a mark and nobody loses one.
Is it fairer to give everyone the mark or to remove the question?
Crediting everyone takes nothing from students who answered correctly. Removal lowers their percentage, which is why Tavakol and Doody (2015) advise against it.
Does EdProcto re-mark the exam if I change the key?
When you ask it to. Correct the key in the builder and press "Re-mark this quiz" on the results page: every attempt handed in is marked again from the key as it stands. The page for universities sets out what it does for a course team.
References
- American Board of Physical Medicine and Rehabilitation. (2017, June 13). Why exam results take weeks to release. https://www.abpmr.org/NewsCenter/Detail/why-exam-results-take-weeks
- Downing, S. M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences Education, 10(2), 133–143. https://doi.org/10.1007/s10459-004-4019-5
- Rudolph, M. J., Daugherty, K. K., Ray, M. E., Shuford, V. P., Lebovitz, L., & DiVall, M. V. (2019). Best practices related to examination item construction and post-hoc review. American Journal of Pharmaceutical Education, 83(7), Article 7204. https://doi.org/10.5688/ajpe7204
- Rush, B. R., Rankin, D. C., & White, B. J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16, Article 250. https://doi.org/10.1186/s12909-016-0773-3
- Srisomsak, V., Sitticharoon, C., Keadkraichaiwat, I., Meethes, S., & Inpaen, I. (2025). Detection of flawed multiple-choice questions in preclinical medical education using item difficulty and discrimination indices: A six-year analysis. BMC Medical Education, 26, Article 92. https://doi.org/10.1186/s12909-025-08204-5
- Tarrant, M., & Ware, J. (2008). Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Medical Education, 42(2), 198–206. https://doi.org/10.1111/j.1365-2923.2007.02957.x
- Tavakol, M., & Doody, G. A. (2015). Making students' marks fair: Standard setting, assessment items and post hoc item analysis. International Journal of Medical Education, 6, 38–39. https://doi.org/10.5116/ijme.54e8.86df
- University of Tennessee System. (2025). COM-G06: Rekey guidelines for preclinical exam questions. https://policy.tennessee.edu/procedure/com-g06-rekey-guidelines-for-preclinical-exam-questions-com-medical-education/
- Young, M., Cummings, B.-A., & St-Onge, C. (2017). Ensuring the quality of multiple-choice exams administered to small cohorts: A cautionary tale. Perspectives on Medical Education, 6(1), 21–28. https://doi.org/10.1007/s40037-016-0322-0