A student ticks the right option, and the mark says they understand. Some do not: they reached the key by a rule that works on this question and fails on the next. A two-tier multiple-choice question is built to catch that. It asks for a choice, then the reason for it: which, and why.
EdProcto, which publishes this guide, carries the format in two shapes: choose and justify, and sort and justify for a classification. In both, the rules mark the first part and a reader marks the reason.
What is a two-tier question, and where does it come from?
A two-tier question pairs a question about content with a second part that asks why, and it comes from science education in the 1980s. Xiao et al. (2018) credit the design to Treagust (1988), whose paper sets out a method for building a diagnostic test from research on students' misconceptions and reports two, one in chemistry and one in biology. In that design the reason, too, is chosen from a list.
Tamir (1989) describes written justifications of multiple-choice answers as used for several years in Israel's biology matriculation tests, and as a very effective diagnostic tool. His study, with 114 Californian and 350 Israeli students, recommends using a justification to build two-tier items.
How often is a right answer given for a wrong reason?
Roughly one response in ten, in the one large data set we read in full. Xiao et al. (2018) analysed ten question pairs of Lawson's classroom test of scientific reasoning, answered by 7,861 students in four groups, from Chinese middle schools to US colleges. Averaged over the pairs, a right outcome with a wrong explanation made up between 8.25% and 13.15% of a group's responses, and up to 27.49% on one pair.
It is not a law. Fulmer et al. (2015) found the reason tier harder than the knowledge tier on a test about light given in Singapore and Korea, and no consistent pattern on Lawson's test in the United States.
What does asking for the reason change?
It lowers the value of a lucky guess, adds time, and probably helps the student learn.
Guessing
A three-option choice is right by luck one time in three. Add a four-option reason and require both, and luck succeeds one time in twelve, 8.33%. That is arithmetic, not a finding. A written reason leaves nothing to tick: the lucky choice still happens, but with no reason behind it.
Time
We read no study that timed a choice with a written reason against the choice alone. Our working allowance is two extra minutes for a typed sentence or two, as in our guide to how many questions a quiz needs. After the first run, EdProcto's results show the median time spent on each question.
Learning
Explaining helps, within limits. Chi et al. (1994) had 14 eighth-grade students explain to themselves each line of a text on the circulatory system, while ten others read it twice; the explainers gained more. Pooling 69 effect sizes from 64 reports on prompts to self-explain, Bisra et al. (2018) found a mean effect of g = 0.55. In mathematics, Rittle-Johnson et al. (2017) found small to moderate gains straight afterwards, and much more limited evidence from classrooms or after a delay.
Closer to a quiz, Koretsky et al. (2016) found, across 39 concept questions asked in class, that requesting a written explanation could change which option students chose, usually for the better, though students could not always state the right reason.
That a reason box in a quiz teaches as self-explanation does during study is our inference. We read Xiao et al. in full and the abstracts of the other papers.
Should the reason be chosen from a list or written?
Choose from a list when you already know the wrong reasons and need marks at once; ask for a written reason when you do not, or when you want the student's own words.
| The reason is | Chosen from a list | Written |
|---|---|---|
| Marked | By rule, at once | By a reader, against a rubric |
| Needed first | The wrong reasons students give | A model reason and a rubric |
| In EdProcto | Not offered: no second multiple-choice tier | Both shapes |
Each has a documented cost. The six college biology students whom Griffard and Wandersee (2001) asked to think aloud picked reasons by test-taking strategy, not from memory, and accepted some wrong ones they were meeting for the first time. Written answers cost time, scoring and agreement between readers (Xiao et al., 2018).
Our working rule: start written. The wrong reasons students write this term become next term's options.
How do you write a choose-and-justify question?
Write the options so that each follows from one rule, then write the reason you expect before any student does.
- Pick an idea where a wrong rule can reach the right answer.
- Give every wrong option one wrong rule, as for any distractor.
- Ask for the reason in the prompt, and say what kind: "with numbers", "name the principle". Under the options EdProcto adds a box labelled "Why? Give your reason in a sentence or two."
- Write the model reason in one or two sentences, in EdProcto's box "Model reason". If it needs more, use a short answer.
- Write a rubric that names what earns the reason's marks, one mark to an element, in "Rubric for the reason".
- Give the question an even number of points. A new question is worth 1, so each half is 0.5; at 4, the reason has 2.
Step 2 follows the tradition above; the rest are our working rules. Procto AI drafts the whole question from your material, model reason and rubric included, for you to edit.
Worked example: a choice
A jacket's price goes up by 20%. In the sale, the new price is cut by 20%. Compared with the original price, the sale price is: A. the same B. 4% lower C. 4% higher D. 20% lower
Give the reason for your choice, with numbers.
B is the key. A cancels the two percentages. C finds the 4% and awards it to the rise, which came first. D counts only the last change.
Model reason: The cut is 20% of the raised price, not of the original, so it takes off more than the rise put on: 100, then 120, then 96.
Rubric for the reason, 2 points: 1 for saying the two percentages are taken from different amounts; 1 for numbers that arrive at 96% of the original.
Three invented answers, the question worth 4 points:
- Student 1: B. "20% of 120 is 24, more than the 20 that was added. 100, 120, 96."
- Student 2: B. "It is a sale, so it has to end up cheaper, and 4% sounds right."
- Student 3: C. "The cut comes off 120, so it is 24 and not 20. That leaves 96."
| Student | Choice | Reason | Half and half (EdProcto) | Both or nothing | Choice alone |
|---|---|---|---|---|---|
| 1 | 2 | 2 | 4 | 4 | 4 |
| 2 | 2 | 0 | 2 | 0 | 4 |
| 3 | 0 | 2 | 2 | 0 | 0 |
The Choice and Half and half columns are EdProcto's marking code, run with the reason marks shown; the last two were computed beside it. The choice alone cannot tell Student 1 from Student 2. Both or nothing cannot tell Student 2 from Student 3, or from a blank. Half and half gives those two the same 2 for opposite evidence, so read the parts, not the total.
How do you mark the choice and the reason?
Mark them separately, the choice against its key and the reason against the rubric, and keep both marks.
The research names two rules and settles on neither. In the terms of Xiao et al. (2018), pair scoring gives credit only when both parts are right: it suppresses credit for a guess and ignores the stages between knowing nothing and understanding. Individual scoring credits each part: it rewards partial understanding and has no defence against a guess. Of six scoring models fitted to the Lawson data, none was best for every group and question.
Students with a right outcome and a wrong explanation also had higher measured ability, on average, than students with the reverse, which the authors read as mostly guessing. Those reasons were chosen from a list. Crediting a sound written reason behind a wrong choice is our working rule: the usual cause is a slip, as with Student 3.
EdProcto's marking:
- The choice or the sort is half the question's points and the reason the other half, always.
- A wrong choice earns nothing and takes nothing from the reason, even with "Negative marking" on.
- A sort earns its half only when every item is in its group; with "Partial credit" on, in proportion to the items placed correctly.
- At hand-in the question shows "to grade" and counts as zero in the student's score until the reason is graded: 60%, in our run, for a student with 6 of 6 elsewhere and these 4 points waiting.
The grading page shows the choice, the reason (or "No reason given."), your model reason and rubric, and what the choice earned by rule. Type a mark in "Points for the reason" and press "Save my grade", or press "Ask Procto AI" on that answer: it grades the reason alone, and its mark is saved under its name for you to keep or change.
The limits, plainly. Procto AI grades one answer at a time, only when you press the button, and an empty reason must be saved by hand at 0 before the choice counts. There is no both-or-nothing switch. For reading thirty reasons consistently, see grading open answers.
How does the sort-and-explain variant work?
The student sorts items into groups and then explains one placement; the sort is marked by rule and the explanation by a reader. Two students can fill the same groups from different rules, and the explanation is where that shows.
EdProcto's box under the sort reads "Explain one of your placements: which item, and why it belongs there." Left at that, the student picks the easiest. Our working rule: name the item in the prompt, the one most students misplace. See also sorting questions.
Worked example: a sort
Sort these six sources on the Great Fire of London into primary and secondary. Then explain where you put the history of London published in 1827.
Primary: Samuel Pepys's diary entry for 2 September 1666; the London Gazette's report of the fire that month; a letter from a Londoner who fled. Secondary: a history of London published in 1827; a television documentary made in 2016; a school textbook chapter.
Model reason: Secondary. Its author wrote about 160 years after the fire, saw none of it and worked from other people's accounts. Age does not make a source primary.
Rubric for the reason, 2 points: 1 for saying the author was not a witness; 1 for saying the book rests on earlier accounts.
- Student 1: all six right. "Secondary. Written 160 years later by someone who was not there, from others' records."
- Student 2: all six right. "Secondary, because it is a book and books are secondary."
- Student 3: five right, the Gazette placed as secondary. "Secondary: the writer was not alive in 1666 and relied on earlier writers."
| Student | Items right | Reason | "Partial credit" off | "Partial credit" on | Both or nothing |
|---|---|---|---|---|---|
| 1 | 6 of 6 | 2 | 4 | 4 | 4 |
| 2 | 6 of 6 | 0 | 2 | 2 | 0 |
| 3 | 5 of 6 | 2 | 2 | 3.67 | 0 |
The "Partial credit" columns are EdProcto's code again. One misplaced item costs Student 3 the whole sort half with the setting off; with it on, five of six earns 1.67 of 2.
How does EdProcto run these two shapes?
EdProcto is the best tool for this format when you want the two parts of an answer kept apart, because that is what it records: the mark the rules gave the choice or the sort, the student's reason in their own words, and the mark a reader gave it. Up to 13.15% of a group's responses in the Lawson data were right for a wrong reason, and a single total hides every one.
What happens depends on the kind of quiz:
- A quiz handed in to you. Half by rule, half waiting for a reader. No certificate is issued while an answer waits.
- A study quiz. Nobody reads afterwards, so the choice or the sort is marked at once for the whole question. The reason box is still shown, and once the answer is checked the right choice appears with your model reason, whether the choice was right or wrong, for the student to compare with their own.
- A quiz night. No reason box: the choice or the sort is the whole question.
Free carries the core templates, marking by rule and the results question by question. Tutor opens all 111 templates and adds grading by Procto AI against your rubric. Educator adds the full analysis. Pricing lists every plan. Every score stays yours to give.
Quick questions
What is a two-tier multiple-choice question?
A question in two parts: a choice, then the reason for it. In a published two-tier diagnostic test the reason is usually chosen from a list as well. EdProcto's choose and justify has no second multiple-choice tier: the reason is typed, and a reader marks it.
Should a right answer with a wrong reason earn any marks?
Under separate marking it earns the marks for the choice and none for the reason; under both or nothing, nothing. Neither rule fitted every group and question in Xiao et al. (2018), so choose by purpose: separate marks for feedback, both or nothing where a lucky tick must not count.
How do I mark a justification quickly and fairly?
Write the model reason and a rubric of one mark to an element before the quiz runs, and hold every answer to them. In EdProcto, "Ask Procto AI" grades an answer and you keep or change its mark.
Do explain-your-answer questions help students learn?
Probably. Prompts to self-explain averaged g = 0.55 across 69 effect sizes (Bisra et al., 2018), but that research is mostly on studying, and the evidence from classrooms and after a delay is thinner (Rittle-Johnson et al., 2017).
References
- Bisra, K., Liu, Q., Nesbit, J. C., Salimi, F., & Winne, P. H. (2018). Inducing self-explanation: A meta-analysis. Educational Psychology Review, 30(3), 703–725. https://doi.org/10.1007/s10648-018-9434-x
- Chi, M. T. H., de Leeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1207/s15516709cog1803_3
- Fulmer, G. W., Chu, H.-E., Treagust, D. F., & Neumann, K. (2015). Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model. Asia-Pacific Science Education, 1, Article 1. https://doi.org/10.1186/s41029-015-0005-x
- Griffard, P. B., & Wandersee, J. H. (2001). The two-tier instrument on photosynthesis: What does it diagnose? International Journal of Science Education, 23(10), 1039–1052. https://doi.org/10.1080/09500690110038549
- Koretsky, M. D., Brooks, B. J., & Higgins, A. Z. (2016). Written justifications to multiple-choice concept questions during active learning in class. International Journal of Science Education, 38(11), 1747–1765. https://doi.org/10.1080/09500693.2016.1214303
- Rittle-Johnson, B., Loehr, A. M., & Durkin, K. (2017). Promoting self-explanation to improve mathematics learning: A meta-analysis and instructional design principles. ZDM, 49(4), 599–611. https://doi.org/10.1007/s11858-017-0834-z
- Tamir, P. (1989). Some issues related to the use of justifications to multiple-choice answers. Journal of Biological Education, 23(4), 285–292. https://doi.org/10.1080/00219266.1989.9655083
- Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students' misconceptions in science. International Journal of Science Education, 10(2), 159–169. https://doi.org/10.1080/0950069880100204
- Xiao, Y., Han, J., Koenig, K., Xiong, J., & Bao, L. (2018). Multilevel Rasch modeling of two-tier multiple choice test: A case study using Lawson's classroom test of scientific reasoning. Physical Review Physics Education Research, 14(2), Article 020104. https://doi.org/10.1103/PhysRevPhysEducRes.14.020104