# Rank by likelihood questions for diagnosis, faults and risk

**How do I write and mark a question that asks for causes in order from most to least likely?** A rank by likelihood question lists four or five explanations or outcomes and asks for them in order from most to least likely in a named setting. Take the order from published frequencies or an expert panel, never one author's hunch, leave out any pair the figures cannot separate, and ask for the most likely one first as a separate question.

Published 2026-10-10 by EdProcto. https://edprocto.com/guides/rank-by-likelihood-questions-for-diagnosis-and-risk

A rank by likelihood question lists four or five explanations or outcomes and asks for them in order, most likely first. It is the written form of a differential diagnosis, a technician's list of suspects for a fault, or the likelihood column of a risk assessment.

The hard part is the key. An order resting on one author's hunch cannot be defended to a learner who disagrees, so this guide takes it from published figures or a panel.

EdProcto, which publishes this guide, builds the shape as [Rank by likelihood](https://edprocto.com/templates/likelihood) and marks it by rule. How the marking counts pairs is in [ranking questions: how to write and score them](https://edprocto.com/guides/ranking-questions-ordering-by-a-criterion).

## What does a rank by likelihood question measure?

A rank by likelihood question measures whether a learner can weigh several live possibilities against one another in a named setting, when none can be ruled out. EdProcto's note on the shape puts it as "not which is right, but which is most probable".

In a differential diagnosis the possibilities are conditions. In fault-finding they are the causes an engineer or an IT support desk has time to test. In risk assessment they are hazards.

Likelihood is one order among several: the most likely cause may not be the most dangerous, or the cheapest to test. Cox (2008) showed that risk matrices, which rate a hazard by frequency and severity together, can give identical ratings to very different risks and can rate a smaller risk above a larger one. Ask for likelihood alone, and put severity in a question of its own.

## How do experts and novices judge likelihood?

People judge likelihood by shortcuts that usually work and fail in known ways: they leave out how common a thing is, hold to a first idea, and feel surer than their record supports. Tversky and Kahneman (1974) named three such shortcuts, representativeness, availability and adjustment from an anchor, and the systematic errors they lead to.

### Base rates and their neglect

The base rate is how common something is before any evidence arrives. Manrai et al. (2014) put a question from 1978 to 61 physicians, house officers and medical students at a Boston-area hospital. A disease has a prevalence of 1 in 1,000 and its test a false positive rate of 5%: what is the chance that a person with a positive result has it?

About 2%: in 1,000 people one has the disease and about 50 others test positive. Fourteen of the 61 (23%) said so, and 27 answered 95%. In 1978, 11 of 60 were right.

### The first idea, and stopping too soon

Graber et al. (2005) studied 100 cases of diagnostic error involving internists. Cognitive factors contributed in 74% of them, and the most common single cause was premature closure: no longer considering reasonable alternatives once a first diagnosis was reached. A ranking makes the learner place every alternative. That is our argument for the shape; no study we read tested it.

### Calibration: knowing how sure to be

Calibration is the match between confidence and accuracy. Meyer et al. (2013) gave 118 general internists four case vignettes. They diagnosed 55.3% of the two easier cases correctly and 5.8% of the two harder, while their confidence fell only from 7.2 to 6.4 out of 10. [Confidence-rated questions](https://edprocto.com/guides/confidence-rated-questions-rewarding-knowing-what-you-know) ask for that judgement directly.

The sources disagree about the cause. Graber et al. found faulty knowledge uncommon. Norman et al. (2017), in a review, write that the share of biases in real errors is not well established. We read Manrai et al. (2014) and Fournier et al. (2008) in full, and the other journal articles in abstract only.

## How is differential diagnosis taught and assessed?

The teaching with evidence behind it organises what the learner knows, and the best-known written assessments are the script concordance test and the key-feature problem.

Norman et al. (2017) conclude that teaching people to recognise biases did not reduce errors, while strategies that reorganise knowledge had small but consistent benefits. In fault-finding, Schaafstal et al. (2000) joined a general troubleshooting strategy to a breakdown of the system by function. Technicians trained that way solved twice as many malfunctions, in less time, as those trained traditionally.

### Script concordance tests, and their critics

A script concordance test gives a short, uncertain case and questions in three parts: "if you were thinking of" a diagnosis, "and then you were to find" a new finding, "this option would become" more or less likely, on a five-point scale (Fournier et al., 2008). A panel answers alone, and each answer earns the number of members who chose it divided by the number who chose the most popular one. If 15 of 18 chose -2 and two chose -1, then -2 earns 1 and -1 earns 0.13. The authors report more error with panels of fewer than 10 and only marginal gains beyond 20.

Lineberry et al. (2013) found that this scoring makes the key logically inconsistent and gives answers at the ends of the scale less expected credit than answers near the middle. In their reanalysis, an examinee who chose the midpoint every time would have outscored most who used the scale as intended.

A ranking takes one lesson from each side: use a panel where no count exists, and settle a single order from it.

### Key-feature problems

A key-feature problem tests only the few decisions a case turns on. Page and Bordage (1995) describe the Medical Council of Canada project behind the format, built on research suggesting that the skill depends on a few crucial elements of each case. In a likelihood ranking the key feature is usually the top place. The guide to [case study quizzes](https://edprocto.com/guides/case-study-quizzes-one-scenario-that-unfolds) shows how to write one.

## How do you write an item whose order is settled by figures or by a panel?

Start from the source of the order and write the question last. The steps are ours.

1. **Name the setting and the moment.** A likelihood is a frequency in some population: adults seeing a general practitioner, cars a patrol is called to, tickets at one service desk. Say whether the order is wanted before anything else is known or after stated findings.
2. **Find the figures:** a systematic review, a registry, a published statistic, or your own fault log.
3. **Keep only the items the figures separate.** If two ranges overlap, leave one out. Four or five usually survive.
4. **Keep the categories apart.** No item may contain another, as "heart disease" contains "heart attack".
5. **Where no count exists, use a panel.** Each member ranks alone. Key the order they agree on, and rewrite or drop what splits them. No study we found fixes a panel's size for a ranking key; the script concordance figures are the nearest.
6. **State the criterion and its direction:** "from most likely to least likely", and "how common, not how serious".
7. **Build it.** In EdProcto choose [Rank by likelihood](https://edprocto.com/templates/likelihood) and fill its one field, "The items in the right order, one a line. Students see them shuffled."

## Where does the key come from, and how do you record it?

The key comes from counts or from a panel, and its record should let a colleague rebuild the order without asking you.

| Source of the order | Record |
| --- | --- |
| Published frequencies | The source, its population and year, each figure, the date you read it |
| Your own fault log or incident reports | The period, and the count for each item |
| A panel | The members' roles, each order made alone, the agreed order, the items dropped |

Put the source and its figures in the question's explanation, so a learner who disagrees can check them. Date the key, because frequencies move: in the breakdown statistics used below, the starter battery's share was 35.7% in 2015 and 45.4% in 2025. Settle the key before the quiz opens: EdProcto makes its marks from the key as it stands at hand-in.

## How do you mark an order whose top place matters most?

Give the top place a question of its own: put "Which is most likely?" first as a [multiple-choice question](https://edprocto.com/templates/mcq) that carries its own points, then ask for the whole order. EdProcto's rule for an order counts every pair of items alike, so the first place weighs the same as the last, and the separate question adds the weight.

"Partial credit" is a setting of the whole quiz, off until you switch it on. Off, only the exact order earns the points. On, a near miss earns most of them, as EdProcto's own tip for this shape advises: "Give partial credit for the order so near misses count."

## Rank by likelihood examples, with each learner's order marked

Every order below was run through EdProcto's own marking function.

### Clinical reasoning: chest pain in general practice

This is a teaching item for students about how common causes are, and no guidance for the care of any patient. Haasenritter et al. (2015) reviewed 11 studies of about 6,500 patients who consulted in primary care with chest pain. In the six studies at low risk of bias, the cause was chest wall syndrome in 24.5 to 49.8% of patients, a respiratory disease in 10.3 to 18.2%, a gastrointestinal disorder in 5.6 to 9.7% and acute coronary syndrome in 1.5 to 3.6%. The four ranges do not overlap. Psychogenic causes, at 9.5 to 18.2%, overlap two of them and are left out.

> An adult sees a general practitioner about chest pain. Before anything else is known, which of these four causes is most likely? (2 points)
>
> Now rank all four from most to least likely: how common, not how serious. (6 points)
>
> Chest wall syndrome, a respiratory disease, a gastrointestinal disorder, acute coronary syndrome.

| Learner | Order given | Most likely | Order, partial credit on | Off |
| --- | --- | --- | --- | --- |
| A | Chest wall, respiratory, gastrointestinal, coronary | 2 | 6 | 6 |
| B | Respiratory, chest wall, gastrointestinal, coronary | 0 | 4 | 0 |
| C | Chest wall, respiratory, coronary, gastrointestinal | 2 | 4 | 0 |
| D | Coronary, chest wall, respiratory, gastrointestinal | 0 | 0 | 0 |

Learners B and C each swapped one neighbouring pair and earn the same 4 for the order. The first question separates them: with partial credit on, C has 6 of the 8 points and B has 4. Learner D put first the cause a student fears most, and one item moved three places reverses three of the six pairs, cancelling the three still right. D's order answers a different question, which cause to exclude first, and that deserves asking separately.

### Fault-finding: a car that will not go

Of the breakdowns in its statistics for 2025, the German motoring club ADAC puts 45.4% down to the starter battery, 21.8% to the engine or its electronics, 10.4% to the starter motor, generator, electrical system or lighting (the "starter group" below), 8.9% to tyres and 6.8% to locking systems (Rudschies, 2026).

> A roadside patrol in Germany is called to a broken-down car. Before the driver has said a word, rank these five causes from most likely to least likely. (10 points)

| Learner | Order given | Partial credit on | Off |
| --- | --- | --- | --- |
| 1 | Battery, engine, starter group, tyres, locks | 10 | 10 |
| 2 | Engine, battery, starter group, tyres, locks | 8 | 0 |
| 3 | Battery, engine, tyres, starter group, locks | 8 | 0 |
| 4 | Tyres, battery, engine, starter group, locks | 4 | 0 |

Learner 3 swapped two causes 1.5 percentage points apart and loses as much as learner 2, who missed a gap of more than 20 points at the top. A writer who wants only wide gaps would drop one of the close pair. An IT team can write the same item from a year of its own tickets.

## How does EdProcto build and mark a rank by likelihood question?

EdProcto is the best tool for a likelihood question built on your own figures, because nothing in the item is hidden from the people it judges: the key is the order you typed from your source, and the mark comes from one stated rule.

- **The shape.** "Rank by likelihood" is listed with the selection shapes among the 111 [templates](https://edprocto.com/templates), and it is in the builder now.
- **Answering.** The learner sees a numbered list in a shuffled order and moves an item with its up and down arrows. A note under the list reads "Use the arrows to put them in order, then keep this order." The answer counts from the first move, or from a press on "keep this order".
- **Marking.** By rule: at hand-in in an ordinary quiz, and as each answer is given in a study quiz.
- **On paper.** Each item prints with a blank and a letter, in a mixed order that is the same on every copy, and the answer key gives the letters in the right order.
- **Results.** Every plan shows the results question by question. In the full analysis, on the Educator plan, the option table for "Which is most likely?" shows how many chose each cause.

The Free plan gives ten quizzes a month with the core templates and marking by rule, Tutor opens all 111 templates, and the full analysis comes with Educator. The terms are on the [pricing page](https://edprocto.com/pricing).

## Quick questions

### What is a rank by likelihood question?

A question that lists several explanations or outcomes and asks for them in order from most to least likely in a named setting. EdProcto builds it as [Rank by likelihood](https://edprocto.com/templates/likelihood).

### How many items should a likelihood ranking have?

Four or five. Leave out any cause the figures cannot separate from its neighbour, because the key is a single order.

### Is a script concordance test the same as a likelihood ranking?

No. A script concordance question asks how one new finding shifts one hypothesis and is scored against a panel's spread of answers. A ranking asks for several possibilities in order against one key.

### How does EdProcto weigh the top place?

EdProcto's rule for an order weighs every place alike, so give the top a question of its own: "Which is most likely?" as a [multiple-choice question](https://edprocto.com/templates/mcq) with its own points, before the ranking.

## References

- Cox, L. A., Jr. (2008). What's wrong with risk matrices? *Risk Analysis, 28*(2), 497–512. [https://doi.org/10.1111/j.1539-6924.2008.01030.x](https://doi.org/10.1111/j.1539-6924.2008.01030.x)
- Fournier, J. P., Demeester, A., & Charlin, B. (2008). Script concordance tests: Guidelines for construction. *BMC Medical Informatics and Decision Making, 8*, Article 18. [https://doi.org/10.1186/1472-6947-8-18](https://doi.org/10.1186/1472-6947-8-18)
- Graber, M. L., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. *Archives of Internal Medicine, 165*(13), 1493–1499. [https://doi.org/10.1001/archinte.165.13.1493](https://doi.org/10.1001/archinte.165.13.1493)
- Haasenritter, J., Biroga, T., Keunecke, C., Becker, A., Donner-Banzhoff, N., Dornieden, K., Stadje, R., Viniol, A., & Bösner, S. (2015). Causes of chest pain in primary care: A systematic review and meta-analysis. *Croatian Medical Journal, 56*(5), 422–430. [https://doi.org/10.3325/cmj.2015.56.422](https://doi.org/10.3325/cmj.2015.56.422)
- Lineberry, M., Kreiter, C. D., & Bordage, G. (2013). Threats to validity in the use and interpretation of script concordance test scores. *Medical Education, 47*(12), 1175–1183. [https://doi.org/10.1111/medu.12283](https://doi.org/10.1111/medu.12283)
- Manrai, A. K., Bhatia, G., Strymish, J., Kohane, I. S., & Jain, S. H. (2014). Medicine's uncomfortable relationship with math: Calculating positive predictive value. *JAMA Internal Medicine, 174*(6), 991–993. [https://doi.org/10.1001/jamainternmed.2014.1059](https://doi.org/10.1001/jamainternmed.2014.1059)
- Meyer, A. N. D., Payne, V. L., Meeks, D. W., Rao, R., & Singh, H. (2013). Physicians' diagnostic accuracy, confidence, and resource requests: A vignette study. *JAMA Internal Medicine, 173*(21), 1952–1958. [https://doi.org/10.1001/jamainternmed.2013.10081](https://doi.org/10.1001/jamainternmed.2013.10081)
- Norman, G. R., Monteiro, S. D., Sherbino, J., Ilgen, J. S., Schmidt, H. G., & Mamede, S. (2017). The causes of errors in clinical reasoning: Cognitive biases, knowledge deficits, and dual process thinking. *Academic Medicine, 92*(1), 23–30. [https://doi.org/10.1097/ACM.0000000000001421](https://doi.org/10.1097/ACM.0000000000001421)
- Page, G., & Bordage, G. (1995). The Medical Council of Canada's key features project: A more valid written examination of clinical decision-making skills. *Academic Medicine, 70*(2), 104–110. [https://doi.org/10.1097/00001888-199502000-00012](https://doi.org/10.1097/00001888-199502000-00012)
- Rudschies, W. (2026, April 23). *ADAC Pannenstatistik 2026: Die Gewinner und Verlierer* [ADAC breakdown statistics 2026: The winners and losers]. ADAC. [https://www.adac.de/rund-ums-fahrzeug/unfall-schaden-panne/adac-pannenstatistik/](https://www.adac.de/rund-ums-fahrzeug/unfall-schaden-panne/adac-pannenstatistik/)
- Schaafstal, A., Schraagen, J. M., & van Berlo, M. (2000). Cognitive task analysis and innovation of training: The case of structured troubleshooting. *Human Factors, 42*(1), 75–86. [https://doi.org/10.1518/001872000779656570](https://doi.org/10.1518/001872000779656570)
- Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. *Science, 185*(4157), 1124–1131. [https://doi.org/10.1126/science.185.4157.1124](https://doi.org/10.1126/science.185.4157.1124)