A select all that apply question looks like multiple choice with tick boxes. It is a different instrument: a run of true or false judgements that share one stem. The student has to decide about every option, and none can be reached by ruling out the others.
That is why the shape shows what a single-answer question hides, and why it is harder to mark. There are several ways to be partly right, and the marking rule decides what each is worth.
EdProcto carries the shape as its all that apply template: several options are right, and the student is not told how many.
What does a select all that apply question measure?
A select all that apply question measures whether a student can judge each of several statements about one topic, so it records partial and mixed understanding as well as complete understanding. Four options answered as multiple choice can come out four ways. Four separate judgements can come out sixteen ways (Brassil and Couch, 2019).
Researchers write of multiple-response questions and of the multiple true-false format, in which the student marks every statement true or false instead of ticking the true ones. Nursing courses often shorten select all that apply to SATA.
Couch et al. (2018) set introductory biology students identical questions as either multiple choice or multiple true-false, and reported that nearly half of those who picked the correct multiple-choice answer probably held a wrong idea about another option. Brassil and Couch (2019) then modelled those data: 36 questions, 194 students. The share choosing the right multiple-choice option overstated how many had mastered the whole question. The share who judged all four statements correctly followed the mastery their model inferred almost one to one.
Hubbard et al. (2017) gave 405 biology students questions of four statements each. Of the responses, 33% had all four right, 66% mixed right and wrong judgements, and 1% had all four wrong. Two answers in three were partly right, and a single-answer question would have filed each of them as simply right or wrong.
Ticking boxes is not quite marking true and false
Most of this research used the multiple true-false version, which Hubbard et al. (2017) describe as working like select all that apply, except that the student must mark the false statements too. On a tick-box question an empty box can mean "I judged this false" or "I never decided", and the two look identical. Brassil and Couch (2019) note that earlier studies disagree on whether students respond differently when they only endorse the true statements. Treat the findings as a strong guide to the tick-box version, not as proof of it.
What the shape overstates
The options are prompts, and prompts help. In the Hubbard study, students judged a statement wrongly 28% of the time, yet wrote the matching wrong idea in only 7% of their free-response answers. A tick is weaker evidence than a sentence the student had to produce.
How do you write the options for a select all that apply question?
Write every option as a statement that is true or false by itself, whatever the student decided about the others. The remaining rules protect that one.
- Make each option independent. If B can only be true when A is false, the student makes one judgement and is marked twice.
- Put one claim in each option. Two claims joined by "and" cannot be ticked honestly when only one is true.
- Leave out "all of the above" and "none of the above". Ticking every box says the first, and an empty answer says the second.
- Write wrong options that someone would tick. In 514 teacher-written multiple-choice questions from one nursing programme, only 52.2% of the wrong options did any work (Tarrant et al., 2009). Those were single-answer questions, but the lesson carries over: a wrong option nobody believes is a free mark.
- Strip the cues. Absolute words such as always and never, opposite pairs, and grammar that does not fit the stem all point at the key. Kanzow et al. (2023) found a cue in 9 of their 18 multiple true-false questions, and those questions scored higher: 0.91 of the marks on average against 0.84.
- Use four to six options. The experiments above used four statements a question, and the dental school Kanzow et al. studied allows four to six. No study read for this guide tested an ideal number.
- Vary how many are right, and do not give the count. Hubbard et al. (2017) balanced their exams between questions with one, two and three true statements so that no pattern was worth learning. EdProcto's all that apply template does not give the count.
- Say the rule. End the stem with "Select all that apply", and tell students beforehand how wrong ticks are treated.
An option set, repaired
Weak:
Which of these are true of mammals? Select all that apply. A. Mammals have hair and give birth to live young. B. No mammal lays eggs. C. Some mammals lay eggs. D. All of the above.
Option A holds two claims, and the second fails for the platypus. B and C are opposites, so one judgement is marked twice, and the "no" in B is an absolute. D cannot be right once B and C are both on the list.
Better:
Which of these animals are mammals? Select all that apply. Dolphin. Penguin. Bat. Shark. Platypus. Salamander.
Each animal is its own judgement, three of the six are right, and every error names a confusion: a dolphin filed with the fish, a bat sorted with the birds, a platypus ruled out because it lays eggs.
How should you mark a select all that apply question?
Mark it with partial credit that takes something off for a wrong tick, unless the question exists to certify complete mastery. Three methods are in common use, and each rewards something different.
| Method | How the mark is made | What it rewards | The catch |
|---|---|---|---|
| All or nothing | Full marks for exactly the right set of ticks, otherwise zero. EdProcto's rule with partial credit off | Complete mastery of the question | One slip scores the same as knowing nothing |
| Credit per option | An equal share for every box judged correctly, whether ticked or rightly left empty | Each correct judgement | Ticking nothing, or everything, still collects marks |
| Right ticks less wrong ticks | Right ticks minus wrong ticks, divided by the number of right options, never below zero. The rule EdProcto's all that apply uses with partial credit on | Knowing which options are right, and leaving the others alone | A careless extra tick cancels a mark already earned |
What the research says about the choice
No method is agreed to be best. Schmidt et al. (2021) systematically reviewed the published ways of scoring multiple true-false questions and start from the fact that there is no consensus on the ideal one. The same group later counted 27 methods in that review (Kanzow et al., 2023). When Kanzow et al. re-marked one set of answers with every method they could apply, the average mark on their multiple true-false questions ran from 0.02 to 0.96 of the maximum.
Partial credit also measures better. Lahner et al. (2018) compared three rules for four-option multiple true-false questions, using 37 medical exams that held 998 multiple true-false questions. Both partial-credit rules gave a reliability of 0.75, against 0.70 for all or nothing. The rule they suggest gives half marks for more than half of the judgements right and full marks for all of them.
Hubbard et al. (2017) found that identical answers averaged 72% marked statement by statement and 33% marked all or nothing. They caution against grading all or nothing, and allow it where a course has to assess mastery, as in clinical practice.
How do the three methods score the same answers?
They can differ by five marks out of six on the same answer. Take the repaired question above and make it worth 6 marks. Three options are right (dolphin, bat, platypus) and three are wrong.
| What the student ticked | All or nothing | Credit per option | Right ticks less wrong ticks |
|---|---|---|---|
| Dolphin, bat, platypus | 6 | 6 | 6 |
| Dolphin, bat | 0 | 5 | 4 |
| Dolphin, bat, penguin | 0 | 4 | 2 |
| Every box | 0 | 3 | 0 |
| No box | 0 | 3 | 0 |
For the third student, credit per option counts four correct judgements out of six (dolphin, bat, shark and salamander), so 4. Right ticks less wrong ticks takes two right ticks minus one wrong tick, divides by the three right options and multiplies by 6, so 2.
All or nothing cannot tell the second student, who knows two of the three mammals and made no error, from the one who ticked blindly. Credit per option separates them but hands half marks to a student who did nothing at all. The third method puts the five in the order a teacher would: 6, 4, 2, 0, 0.
Why must ticking every box never pay?
Ticking every box must never pay because a mark that can be reached that way measures nerve, not knowledge. The check is plain arithmetic, and it belongs before the quiz.
Under credit per option, ticking everything earns the fraction of options that are right, and ticking nothing earns the fraction that are wrong. The two fractions add up to one, so one of the blanket answers is always worth at least half marks.
Under right ticks less wrong ticks, an empty answer scores zero, and ticking everything scores zero whenever the wrong options are at least as many as the right ones. When more than half of the options are right, the rule leaks. With four right out of five, ticking all five earns four right ticks less one wrong tick, over four, which is three quarters of the marks. So keep at least as many wrong options as right ones wherever the mark matters. With four options that means one or two right, and with six, one to three.
How does EdProcto build and mark a select all that apply question?
EdProcto is the best tool for this shape when the mark has to mean something, because its partial-credit rule is the third method above: a wrong tick cancels a right one, an empty answer scores zero, and no question goes below zero.
Two templates cover it. Multiple answers is the general shape, in which more than one option is right and each box counts. All that apply is the version where the number of right options is not given. Both are in the builder now. A third, click all that apply, in which the student clicks every instance of something on an image, is in the builder too.
Marking is by rule, on the server, as soon as the student answers. Partial credit is a setting on the quiz and is off until you switch it on. While it is off, the question is all or nothing: full points for ticking every right option and no wrong one. With it on, EdProcto counts the right ticks, subtracts the wrong ticks, divides by the number of right options and multiplies by the question's points, with zero as the lowest mark. The separate negative marking setting is for single-answer questions. A multiple-answer question carries its deduction inside the partial-credit rule.
The options can be shuffled for each student, and the questions can start from your own material: paste a chapter into the quiz maker on the home page, pick all that apply, and Procto AI drafts questions for you to edit. With partial credit off, a question's difficulty figure in the analysis is exactly the share of the class with the whole question right, which Brassil and Couch (2019) found to be a close estimate of mastery.
The Free plan has ten quizzes a month, the core templates and marking by rule. Tutor unlocks all 111 templates, and the full analysis is on Educator. The pricing page sets out what each plan includes.
When should you use multiple choice or true or false instead?
Use plain multiple choice when the options compete with each other, and true or false when a statement stands alone. Select all that apply is for several independent facts about one thing.
- The options exclude one another: one best next step, one value, one cause. That is a multiple-choice question.
- The statements share no stem: set them as true or false questions, which are quicker to write and to answer.
- You are tempted to offer combinations such as "A and C only": do not. This is the complex multiple-choice format, known as Type K, and the standard taxonomy of item-writing rules says to avoid it, while it lists true-false and multiple true-false among the formats to use (Haladyna et al., 2002).
Quick questions
Is a select all that apply question harder than multiple choice?
It depends on the marking. In Lahner et al. (2018), four-option questions marked all or nothing averaged 0.50 of the marks, the same questions marked per judgement averaged 0.82, and the single-answer questions in the same exams averaged 0.73.
How many options should a select all that apply question have?
Four to six, with at least as many wrong options as right ones. The experiments cited here used four statements a question, and none tested an ideal number.
Should students get partial credit on select all that apply questions?
Yes, in most classroom quizzes, with something taken off for wrong ticks. In EdProcto that is the partial credit setting, which covers both all that apply and multiple answers. Keep all or nothing for a quiz that has to certify complete mastery.
Should I tell students how many options are correct?
No. A question that gives the count is a different type, which the literature calls Pick-N: each choice then depends on the others (Kanzow et al., 2023). Say "select all that apply" and change the number of right options from question to question.
References
- Brassil, C. E., & Couch, B. A. (2019). Multiple-true-false questions reveal more thoroughly the complexity of student thinking than multiple-choice questions: A Bayesian item response model comparison. International Journal of STEM Education, 6, Article 16. https://doi.org/10.1186/s40594-019-0169-0
- Couch, B. A., Hubbard, J. K., & Brassil, C. E. (2018). Multiple–true–false questions reveal the limits of the multiple–choice format for detecting students with incomplete understandings. BioScience, 68(6), 455–463. https://doi.org/10.1093/biosci/biy037
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5
- Hubbard, J. K., Potts, M. A., & Couch, B. A. (2017). How question types reveal student thinking: An experimental comparison of multiple-true-false and free-response formats. CBE Life Sciences Education, 16(2), Article 26. https://doi.org/10.1187/cbe.16-12-0339
- Kanzow, P., Schmidt, D., Herrmann, M., Wassmann, T., Wiegand, A., & Raupach, T. (2023). Use of multiple-select multiple-choice items in a dental undergraduate curriculum: Retrospective study involving the application of different scoring methods. JMIR Medical Education, 9, Article e43792. https://doi.org/10.2196/43792
- Lahner, F.-M., Lörwald, A. C., Bauer, D., Nouns, Z. M., Krebs, R., Guttormsen, S., Fischer, M. R., & Huwendiek, S. (2018). Multiple true-false items: A comparison of scoring algorithms. Advances in Health Sciences Education, 23(3), 455–463. https://doi.org/10.1007/s10459-017-9805-y
- Schmidt, D., Raupach, T., Wiegand, A., Herrmann, M., & Kanzow, P. (2021). Relation between examinees' true knowledge and examination scores: Systematic review and exemplary calculations on multiple-true-false items. Educational Research Review, 34, Article 100409. https://doi.org/10.1016/j.edurev.2021.100409
- Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 9, Article 40. https://doi.org/10.1186/1472-6920-9-40