The quiz is marked, and on one question sixteen of thirty students chose the same wrong answer. The mark says only that most of the class missed it. Distractor analysis says which wrong answer they chose, and who chose it.
Examiners call wrong options distractors. EdProcto, which publishes this guide, counts the takers of every option in the full analysis of its results and sets their average score beside the count. The item analysis guide introduces every figure on that page; this one stays with the options.
What is distractor analysis, and what are its four patterns?
Distractor analysis is the reading of two figures for each option of a multiple-choice question: how many students chose it, and how those students did on the whole quiz. The first says how attractive the option was, the second to whom.
A score hides both: on a question 40% got right, the wrong answers may be spread evenly, which is guessing, or piled on one option, which is a belief. Almost every wrong option fits one of four patterns.
| Pattern | Usual meaning | This week |
|---|---|---|
| Nobody chose it | Never plausible, or an error the class has outgrown | Rewrite the option |
| It drew weaker students | Working as intended | Leave alone |
| It drew stronger students | A second right answer, or a wrong key | Rewrite, or re-key |
| Most of the class chose it | A shared misconception, or something taught wrongly | Reteach |
Weaker and stronger mean scoring lower or higher on the same quiz than the students who chose the key.
The third pattern has two causes. If the popular "wrong" option is simply right, the key was set on the wrong option: the question with the wrong key has the signs and the check. If both can be defended, the stem is ambiguous.
In the fourth, the takers score below the key's: they share an idea, and sometimes it came from the lesson.
What does research report about wrong options?
Three things: many wrong options do no work, an option is best judged by who chose it, and options built on known errors show what a class believes. Each source was read on 10 October 2026: Haladyna and Downing, DiBattista and Kurzawa, Attali and Fraenkel, Herrmann-Abell and DeBoer, and Wind et al. in abstract only.
The 5% line, and how many options pass it
The usual rule calls a distractor non-functioning when under 5% of students choose it. Forthmann et al. (2020), a paper on methods, credit it to Haladyna and Downing (1993), whose abstract concludes, from three testing programmes, that items seldom hold more than three useful options. Tarrant et al. (2009) call under 5% the usual definition. Nothing we read derives the figure: it is a convention.
DiBattista and Kurzawa (2011) analysed 1,198 questions from sixteen undergraduate classroom tests in various disciplines. Of 3,819 distractors, 45% were flawed: chosen by under 5% of students, or chosen more by high scorers than by low. The second fault was the costly one: a question separated students far less well when a distractor drew the stronger ones, and hardly less for one that few chose.
Reading options by score group: the trace line
A trace line plots, for one option, the share of students choosing it against their total score. Forthmann et al. (2020) trace it to Levine and Drasgow in 1983 and to Wainer, who named it in 1989: split the students into score groups, five in their example, and draw a line for each option. In their first example the key's line rises with each group and every distractor's falls.
One class cannot fill five groups: Tarrant et al. (2009) note that trace lines typically need 200 or more examinees. The classroom version is one number for each option, the mean total score of its takers, set beside the key's. That is the comparison Attali and Fraenkel (2000) proposed, as Forthmann et al. describe it: a distractor's takers against the key's, not against everyone else, which their abstract says can wrongly reject a question.
What options built on misconceptions add to teaching
When each wrong option carries one known wrong idea, the counts map what the class believes. Herrmann-Abell and DeBoer (2011) gave 91 such chemistry questions to 13,360 students from middle school to college across the USA. Curves of each option's probability against overall performance showed, in some cases, misconceptions fading one after another as performance rose, and on one question a distractor's odd curve exposed a flaw.
Wind et al. (2019) analysed 128 seventh-grade students' answers to 20 questions on the engineering design process. Low, medium and high achievers favoured different options, and on several questions the pattern differed from one teacher's classroom to another's.
So an answer that sweeps a class can belong to the teaching. Neither study measured whether lessons improved afterwards: for that, see finding and fixing misconceptions.
How stable is an option's share in a class of thirty?
Not very: in thirty students an option's count moves by several from class to class by chance alone. The table is our arithmetic, not a finding: an option that truly attracts a fixed share of such students, and thirty of them choosing independently (exact binomial sums).
| True share | Takers in at least 19 classes in 20 | Nobody chooses it | Under the 5% line |
|---|---|---|---|
| 5% | 0 to 4 | 21% | 55% |
| 10% | 0 to 7 | 4% | 18% |
| 20% | 2 to 11 | 0.1% | 1% |
| 33% | 5 to 15 | under 0.1% | under 0.1% |
- An empty row proves little. An option that attracts one student in ten draws nobody in about one class in 24.
- The 5% line is for large samples. In thirty it means two takers, and an option worth exactly 5% misses it more often than not.
- Small leads are chance. Two options that each attract 30% finish four or more students apart in four classes in ten. Sixteen of thirty is no accident: an option worth a third gets there in about one class in 50.
"Their mean" is rougher: with a spread of 20 points across the class, the mean of three takers has a standard error of about 12 points (20 ÷ √3), and it is no steadier than the quiz's reliability.
How do you read the options table in ten minutes?
Read it in this order, before any score goes out.
- Start at the top (1 minute). In EdProcto's results, "What it says about the teaching" names any question with a shared wrong answer or a key to check.
- Open those questions (4 minutes). Under "Each question", press a question's row: a table opens headed "Option", "Chose it" and "Their mean", with "key" beside the right answer.
- Compare the leading wrong option with the key. A mean well below the key's is a misconception. Level or above: answer the question yourself, then check the key.
- Scan the rest for quiet cases (3 minutes). A wrong option with a third of the class while half are right draws no sentence, nor does one with fewer takers than the key and a higher mean.
- Note the empty rows (1 minute). They change nothing this week.
- Write one word beside each question (1 minute): reteach, rewrite, re-key or leave.
What should you do with each pattern the same week?
Match the action to the pattern. The four examples are ours, each on an invented class of thirty.
Reteach: maths
A rectangle is enlarged so that every side is twice as long. How many times larger is its area?
Seventeen chose "2 times" and nine the key, "4 times", and the seventeen scored lower overall: they think area grows in step with length. Next lesson, draw a 2 by 3 rectangle and a 4 by 6 one and count the squares, 6 and 24.
Rewrite: history
In which year did the First World War end?
The key, 1918, drew fourteen. Nine of the strongest students chose 1919, when the Treaty of Versailles was signed. Both can be defended, so credit both, and next year ask when the armistice stopped the fighting on the Western Front.
Re-key: geography
Which city is the capital of Australia?
Twenty-three chose Canberra, the top scorers among them, and four the keyed answer, Sydney. The class is right. In EdProcto, correct the key in the builder and press "Re-mark this quiz" on the results page: every attempt handed in is marked again from the corrected key.
Leave alone: English
Which word in "The dog barked loudly at the old gate" is an adverb?
Twenty-one chose "loudly" and seven "old", and the seven averaged twenty points below the rest. The option has found the students who confuse adjectives with adverbs: a five-minute group, and a question to keep.
An option nobody chose waits too. Before the next use, replace it with a real error, as in distractors that work, or drop it, as in how many options a question needs.
How do you keep a record of the options across years?
Keep one line for each option of a question you reuse, and add to it each time it is set.
- Copy the option's words, how many chose it and their mean, with the date and the size of the class. EdProcto's spreadsheet file from "Download the marks" holds each student's points on every question; the option counts are on the results page, in the opened row.
- Pool the counts across classes and years before using the 5% line.
- Watch a misconception's share after the reteach. If it sweeps next year's class too, change the lesson, not the question.
A worked example: thirty students, four questions
This class is invented, and each figure is what EdProcto's analysis code gave back for its answers. Thirty students sat ten four-option questions on states of matter, one point each, and averaged 56.3%. Four questions show the four patterns.
| Question and option | Chose it | Their mean |
|---|---|---|
| Q2 key: Gas to liquid | 22 (73%) | 64.1% |
| Q2: Solid to gas | 0 (0%) | – |
| Q4 key: They vibrate faster | 19 (63%) | 67.9% |
| Q4: They get bigger | 8 (27%) | 38.8% |
| Q6 key: Water vapour | 10 (33%) | 71% |
| Q6: Air | 16 (53%) | 53.1% |
| Q8 key: Ice melting | 12 (40%) | 56.7% |
| Q8: Sugar dissolving in water | 13 (43%) | 66.9% |
Q2 asked what condensation is. Beside it the code wrote:
Nobody chose "Solid to gas" on Q2. A wrong answer nobody picks is doing no work; make it plausible.
Q4 asked how a solid's particles change as it warms. "They get bigger" drew eight students 29 points below the key's, and no sentence: working.
Q6 asked what fills the bubbles in boiling water. Beside it:
More students chose "Air" than the right answer on Q6. Either the key is wrong or the class shares a misconception.
The sixteen scored 18 points below the key's ten, so the top of the page read:
A shared wrong answer on Q6: the class was taught something, and it was not this. Worth a minute in the next lesson.
Q8 asked which change is physical, and "More students chose" appeared again. This time the thirteen outscored the key's twelve by ten points, so the top read:
Check the key on Q8: the students who chose the most popular wrong answer scored higher on the rest of the quiz than those who chose the keyed one.
Dissolving is a physical change too: two right answers.
With two students moved, eleven on this answer against the key's thirteen, the code wrote nothing about Q8. The sentences are a floor; the table is the evidence.
How does EdProcto show distractor analysis?
EdProcto is the best tool for reading wrong answers when the lesson is tomorrow, because it does the counting and writes the two patterns that cannot wait, a shared wrong answer and a suspect key, at the top of the page. In the example it told Q6 from Q8 by their takers' means.
What is counted. Every question answered by choosing one option, such as multiple choice, over the full sittings students have handed in. It leaves out your own preview and any retry of the misses. "Chose it" is the number who chose the option and their share of those sittings. "Their mean" is their average score on the whole quiz, this question included, so on ten one-point questions the key's takers start ten points ahead. An answer left blank, or never reached, is counted under no option and still counts in the class, so the shares can add up to under 100%. With "Shuffle the options" on, the table still counts by option, whatever order each student saw.
What triggers each sentence.
- "Nobody chose": a wrong option with no takers, once ten sittings have scores.
- "More students chose": a wrong option with at least three takers, and more than the key has.
- "A shared wrong answer on": a question with that sentence, when nothing points at the key.
- "Check the key on", as quoted for Q8: that option's takers have the higher mean, or nobody chose the key.
A fifth appears whenever "Separates", the discrimination index, is below zero:
Check the key on Q5: the stronger students got it wrong more often than the weaker ones.
The plans. A quiz takes thirty students on Free and 100 on Tutor, where the results show each question's share right and its time. The options table and these sentences are part of the full analysis, on the Educator plan and above, which takes 500 students: see pricing. Start on the home page with any of the templates.
Quick questions
What is a non-functioning distractor?
A wrong option that does no work: by the usual research rule, one chosen by fewer than 5% of students, or chosen more by strong students than by weak (Tarrant et al., 2009).
What does it mean when most students chose the same wrong answer?
If its takers scored below the key's, the class shares a misconception or was taught something wrongly: reteach. If they scored as high or higher, check the key and the wording first.
How many students make a wrong answer common?
No line is agreed. EdProcto writes its sentence at three or more takers and more than the key has; in thirty students, a lead of three or four is within chance.
Where does EdProcto show who chose each option?
In the results, under "Each question": press a question's row. The table is part of the full analysis on the Educator plan: see pricing.
References
- Attali, Y., & Fraenkel, T. (2000). The point-biserial as a discrimination index for distractors in multiple-choice items: Deficiencies in usage and an alternative. Journal of Educational Measurement, 37(1), 77–86. https://doi.org/10.1111/j.1745-3984.2000.tb01077.x
- DiBattista, D., & Kurzawa, L. (2011). Examination of the quality of multiple-choice items on classroom tests. The Canadian Journal for the Scholarship of Teaching and Learning, 2(2), Article 4. https://doi.org/10.5206/cjsotl-rcacea.2011.2.4
- Forthmann, B., Förster, N., Schütze, B., Hebbecker, K., Flessner, J., Peters, M. T., & Souvignier, E. (2020). How much g is in the distractor? Re-thinking item-analysis of multiple-choice items. Journal of Intelligence, 8(1), Article 11. https://doi.org/10.3390/jintelligence8010011
- Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item? Educational and Psychological Measurement, 53(4), 999–1010. https://doi.org/10.1177/0013164493053004013
- Herrmann-Abell, C. F., & DeBoer, G. E. (2011). Using distractor-driven standards-based multiple-choice assessments and Rasch modeling to investigate hierarchies of chemistry misconceptions and detect structural problems with individual items. Chemistry Education Research and Practice, 12(2), 184–192. https://doi.org/10.1039/C1RP90023D
- Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 9, Article 40. https://doi.org/10.1186/1472-6920-9-40
- Wind, S. A., Alemdar, M., Lingle, J. A., Moore, R., & Asilkalkan, A. (2019). Exploring student understanding of the engineering design process using distractor analysis. International Journal of STEM Education, 6, Article 4. https://doi.org/10.1186/s40594-018-0156-x