Case study questions put one realistic situation in front of a student and keep asking what they would do next. The first question comes with the opening facts only. Each later one brings something new: a result, a reply, a change of plan. Nobody can go back.
That is how judgement is used at work, which is why lecturers in nursing, medicine, law, business and engineering turn to a case-based quiz when isolated facts are not enough. EdProcto builds the unfolding case as its case study template, a structure for the whole quiz.
The research, most of it from medical examinations, gives three rules: write the decisions before the story, keep a case to two or three questions, and never rest a grade on one case.
What do case study questions measure?
Case study questions measure whether a student can make the right decision at the right moment on incomplete information, which separate fact questions cannot show.
Schuwirth et al. (2001) had twenty general practitioners and twenty students think aloud over short case-based questions and matched factual ones. The factual questions mostly produced a check of true or false. The cases produced longer reasoning, in which information was weighed for how far it pointed one way or the other.
The scores differ too. Hrynchak et al. (2014) reviewed 21 articles on key-feature questions and found only very modest correlations with general knowledge examinations, a sign that the two do different jobs. Key-feature results have also predicted doctors' later performance.
Two cautions. Nearly all of this evidence is from medicine, so a lecturer in law or engineering should treat the principles as tested and the numbers as borrowed. And the format guarantees nothing: Schuwirth and van der Vleuten (2003) stress that a question's content, far more than its format, decides what it tests. A case whose questions can be answered without the scenario is a recall quiz with a story attached.
What did the research on case-based formats find?
The research found that short cases with a few questions on the critical decisions work, and that the long, lifelike simulation most people picture does not.
Swanson et al. (1987) reviewed dozens of studies of long written and computer-based simulations. Examinees found them realistic, but the scores were too imprecise to interpret unless a test held an impractically large number of them.
In 1986 the Medical Council of Canada, whose qualifying examination then carried a booklet of patient management problems, long written simulations of this kind, commissioned a six-year project to build a more valid written test of clinical decision-making. Page and Bordage (1995) give two reasons: numerous studies did not support the long problems, and research suggested that the skill is specific to the case in hand and depends on a few crucial elements of it, the problem's key features. The key-feature problem is a short scenario with questions on those elements and nothing else. Across the studies reviewed by Hrynchak et al. (2014) its reliability ran from 0.49 to 0.95, and they conclude that the research supports its use.
The finding behind that second reason is called case specificity: success on any one case is specific to that case (Norman et al., 2006). A student who handles one scenario well may handle the next badly, so a score from one long case depends heavily on which case came up.
How do you write a case that unfolds?
Write the decisions first and the story last. In the steps Farmer and Page (2005) give for key-feature problems, the key features are defined before any question is written, and Grumer et al. (2019) wrote each scenario to fit key features already defined.
- Take the problem from your syllabus, not from a good anecdote.
- List the key features: the two or three decisions on which the case turns, above all those your students most often get wrong.
- Write the opening with only what a person would know at the start, in everyday words. Never name the concept.
- Write one question for each decision. Let the decision choose the format: pick one, pick a stated number, or write the answer in. Page et al. (1995) allow all of these.
- Attach each new fact to its question, and say what happened whatever the student chose: "The figures come back", never "If you asked for the figures".
- Read the screens in order for leaks. A later screen may give away an earlier answer. None may give away its own, or one still to come.
- Settle the key and the marks, then ask a colleague to sit the case cold.
- Build it. In EdProcto's case study structure the scenario is held once for the whole quiz and shown before the questions, and each question has room for extra material under its prompt: the place for the next fact.
Expect to be slow at first: Schuwirth and van der Vleuten (2003) write that an inexperienced teacher may need up to three hours for one key-feature case, while practised writers may produce up to four an hour.
Why there is no going back
No going back is what makes step 5 safe. In the electronic key-feature examination of Grumer et al. (2019) each question could be answered once, which allowed the correct answer to be revealed in the following question; students could look back but not edit.
How many questions for a case, and how many cases for a quiz?
Two or three questions for each case, and as many cases as the time allows. Norman et al. (2006) modelled the trade-off on three Medical Council of Canada examinations of about 35 cases each, sat by 6,342 graduating students. For a fixed testing time, reliability was lowest with one question a case and highest with two and three. About 80% of the error came from questions within cases, so piling questions on one scenario buys little.
Grumer et al. (2019) set 12 problems in 60 minutes and report a Cronbach's alpha of 0.53 for 81 students, once three who broke off early were left out. McNamara et al. (2023), citing the format's developers, put the need at up to 40 cases for a coefficient of 0.8.
So one case makes a good class exercise or low-stakes quiz. A grade that matters needs many short cases, spread across the term's quizzes and backed by ordinary questions that cover the syllabus.
How should a case study quiz be scored?
Mark every decision separately, weight the questions within a case equally, and make each case count the same. Grumer et al. (2019) adapted that scheme from Page and Bordage (1995): each question is worth up to one and the questions in a problem are averaged, so every problem is worth one however many questions it holds.
Should a wrong early answer sink the rest?
No. A student who gets the first decision wrong should lose that mark and no other. The reveal in step 5 sees to it: the next screen gives everyone the same facts, so all start the second question level. A quiz where a wrong answer leads down another path is adaptive branching, a different structure in EdProcto's builder.
Calculations are the exception, because the student's own answer carries forward: use a scaffolded sequence with credit at each step, so an early slip does not zero the rest.
What do dependent questions do to reliability?
They inflate the estimate. Questions that share a scenario are not independent evidence: a student who misreads the opening may lose every mark in the case. Keller et al. (2003) state that such context-dependent item sets invariably make the items depend on one another. Sireci et al. (1991) measured the effect on two reading comprehension tests, each built from four sets they call testlets: reliability calculated the traditional way was a substantial overestimate.
No scoring rule removes the problem. Keller et al. (2003) compared marking each question right or wrong with scoring a set as one multi-point question: the first may overstate what the test tells you, the second may understate it. So keep cases short and numerous, and distrust a reliability figure from a quiz built on one case.
A worked case: the gate workshop
This case is ours, from operations management. Its key features came first: find out what each stage can do before judging a purchase; the slowest stage sets the output of the line; relieve that stage and the limit moves.
The scenario, shown before the questions. You are the new operations manager at a firm that makes steel garden gates. Every gate is cut, then welded, then painted. For a month, orders have gone out late. The owner wants to buy a faster cutting machine and asks for your view.
| Step | New information | Question and options | Key |
|---|---|---|---|
| 1 | None | What do you ask for first? The machine's price; each stage's daily capacity and the daily orders; a list of late customers; last month's overtime | Capacities and orders |
| 2 | Cutting can finish 60 gates a day, welding 36, painting 50. Orders average 44. The machine would lift cutting to 90 | With the machine and no other change, how many gates a day will the firm finish? 36, 44, 50 or 90 | 36 |
| 3 | Welding holds the line at 36, so the machine adds nothing. A second welder would lift welding to 55 | What is the most the firm could then finish in a day? 44, 50, 55 or 60 | 50: painting is the new limit |
One mark a question. Where a quiz holds several cases, scale each to the same total. Step 2 hands over the figures whether or not the student asked for them, and step 3 states the answer to step 2, which is no leak when nobody can return. The wrong options are real errors: 90 trusts the machine, 55 forgets that the limit moves.
What are the common mistakes?
Writing the story first is the mistake behind most of the others.
- Questions that do not need the case. Cover the scenario. If the question can still be answered, it belongs in an ordinary quiz.
- Two decisions in one question. A wrong answer no longer says which failed.
- A chain with no reveal. If question three needs a right answer to question two, one slip is punished twice.
- The same case every year. Large numbers of cases are needed to stop students memorising them (Schuwirth & van der Vleuten, 2003).
How does EdProcto run a case study quiz?
EdProcto is the best tool for a case study quiz drawn from your own teaching, because the rule the electronic key-feature examinations depended on is part of the template, not a setting to remember. Grumer et al. (2019) could reveal each answer in the next question only because no answer could be changed. The case study structure rules out two settings for that reason: "Allow going back to change an answer", because the template has no going back, and "Shuffle the questions for each student", because the order of the questions is the structure.
Keep the layout setting at one question a page, the default, so each fact arrives with its own question.
Every question inside the structure takes whichever shape suits its decision and is marked by its own kind: choices by rule at once, written answers by you or, when you ask on an answer, by Procto AI against your rubric. Procto AI writes the questions from your material, and among the kits is a case to work through, for which it writes the scenario too. Choosing the key features remains your job.
Each question carries its own points and the concept it tests. Enter its key feature as the concept and the analysis gives a mean for each, which shows the decision a class missed.
Free covers the core templates and every kit, with ten quizzes a month. Tutor opens all 111 templates and lifts that limit, which a course of many short cases will reach. Educator adds the full analysis. The rest is on the pricing page.
When is a plain multiple-choice or best-and-worst question enough?
A single question is enough when the judgement happens at one moment and no later fact would give the answer away.
| What you need to see | Shape in EdProcto |
|---|---|
| One decision on a short vignette | Multiple choice, written from your material in the quiz maker on the home page |
| Judgement between defensible actions at one moment | Best and worst: the student picks both ends |
| One calculation worked in steps | Scaffolded sequence: each answer carries forward |
| How much information a student needs first | Staged reveal, where each stage revealed costs points |
Inside any structure a question is still a question: the rules in how to write multiple-choice questions that measure apply to each one.
Quick questions
What is a key-feature question?
A key-feature question sits on a short, realistic scenario and asks only for an essential decision, in multiple-choice or open form (Schuwirth & van der Vleuten, 2003). Each targets a critical step in resolving the problem (Page et al., 1995).
How many questions should a case study have?
Two or three. Norman et al. (2006) found reliability for a fixed testing time highest with two and three questions a case and lowest with one.
Can students go back and change an answer in a case study quiz?
They should not, because later screens reveal earlier answers. EdProcto's case study template rules out going back and shuffling the questions.
Can one case study be the whole examination?
Not for a grade that matters. Success on one case predicts the next poorly, so use several short cases and add ordinary questions for coverage.
References
- Farmer, E. A., & Page, G. (2005). A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39(12), 1188–1194. https://doi.org/10.1111/j.1365-2929.2005.02339.x
- Grumer, M., Brüstle, P., Lambeck, J., Biller, S., & Brich, J. (2019). Validation and perception of a key feature problem examination in neurology. PLoS ONE, 14(10), Article e0224131. https://doi.org/10.1371/journal.pone.0224131
- Hrynchak, P., Glover Takahashi, S., & Nayer, M. (2014). Key-feature questions for assessment of clinical reasoning: A literature review. Medical Education, 48(9), 870–883. https://doi.org/10.1111/medu.12509
- Keller, L. A., Swaminathan, H., & Sireci, S. G. (2003). Evaluating scoring procedures for context-dependent item sets. Applied Measurement in Education, 16(3), 207–222. https://doi.org/10.1207/S15324818AME1603_3
- McNamara, L. M., Scott, K. M., Boyd, R. N., Farmer, E. A., Webb, A. E., & Novak, I. E. (2023). Constructing validity evidence from a pilot key-features assessment of clinical decision-making in cerebral palsy diagnosis: Application of Kane's validity framework to implementation evaluations. BMC Medical Education, 23, Article 668. https://doi.org/10.1186/s12909-023-04631-4
- Norman, G., Bordage, G., Page, G., & Keane, D. (2006). How specific is case specificity? Medical Education, 40(7), 618–623. https://doi.org/10.1111/j.1365-2929.2006.02511.x
- Page, G., & Bordage, G. (1995). The Medical Council of Canada's key features project: A more valid written examination of clinical decision-making skills. Academic Medicine, 70(2), 104–110. https://doi.org/10.1097/00001888-199502000-00012
- Page, G., Bordage, G., & Allen, T. (1995). Developing key-feature problems and examinations to assess clinical decision-making skills. Academic Medicine, 70(3), 194–201. https://doi.org/10.1097/00001888-199503000-00009
- Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2003). ABC of learning and teaching in medicine: Written assessment. BMJ, 326(7390), 643–645. https://doi.org/10.1136/bmj.326.7390.643
- Schuwirth, L. W. T., Verheggen, M. M., van der Vleuten, C. P. M., Boshuizen, H. P. A., & Dinant, G. J. (2001). Do short cases elicit different thinking processes than factual knowledge questions do? Medical Education, 35(4), 348–356. https://doi.org/10.1046/j.1365-2923.2001.00771.x
- Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28(3), 237–247. https://doi.org/10.1111/j.1745-3984.1991.tb00356.x
- Swanson, D. B., Norcini, J. J., & Grosso, L. J. (1987). Assessment of clinical competence: Written and computer-based simulations. Assessment & Evaluation in Higher Education, 12(3), 220–246. https://doi.org/10.1080/0260293870120307