Ranking questions: how to write and score them, with examples

A ranking question lists a handful of items and asks for them in order by a criterion the question names: largest to smallest, most to least likely, strongest to weakest evidence. One answer commits the student to a comparison between every pair of items.

EdProcto, which publishes this guide, builds the shape as its ranking template and, for probabilities, as rank by likelihood, and marks both by rule.

What is a ranking question, and where do ranking tasks in physics come from?

A ranking question asks for items of one kind in order along one stated criterion. An ordering question asks what comes before what; a ranking asks which has more of something.

Physics teachers know the form as the ranking task. PhysPort (n.d.), a site of the American Association of Physics Teachers, describes ranking task exercises, developed by Thomas O'Kuma, David Maloney and Curtis Hieggelke: students rank variations of a physical situation by a specified quantity and explain their reasoning. The stated aim is to draw out a student's own ideas about how physical systems behave, not a memorised response.

Maloney (1985) had earlier given science and non-science majors sets of tasks about carts and the conservation of mechanical energy, and found an identifiable strategy in over 97% of cases: a set of variations shows the rule in use.

What studies of ranking tasks found

• A ranking exposes shortcuts. Maeyer and Talanquer (2010) had undergraduate chemistry students rank substances by a physical or chemical property. Many relied on what the authors call recognition, representativeness, one-reason decision making and arbitrary trend, which gave answers without the knowledge needed and often led students astray.
• A right order can come from a wrong reason. In a qualitative study of organic chemistry students ranking compounds by acid strength, McClary and Talanquer (2011) found that shortcuts sometimes gave correct responses, and that very few participants predicted the correct trends from scientifically acceptable arguments.
• Ranking tasks teach. Hudgins et al. (2006) followed 253 astronomy students through eight topics: average multiple-choice scores rose from 32% before instruction to 61% after traditional instruction and 77% after collaborative ranking tasks. It was one group with no comparison class, and it says nothing about ranking as a marked question.

We could not read a study that set one topic both as a ranking and as a calculation. The nearest is Singh (2008), who paired conceptual and quantitative questions on one principle in introductory mechanics. Students given only the conceptual question seldom tried to turn it into a calculation, and those who answered both often did better on the conceptual one. Those were not rankings.

What does a ranking measure that separate questions do not?

A ranking measures comparison along one dimension, and whether the student applies one rule to every item, without asking for a single value. Five items hold ten pairs, and one answer settles all ten. "Which trolley has the most kinetic energy?" tests only the top, and five separate calculations test one substitution five times.

A ranking can be answered with no calculation: double the speed and the energy is four times as large. The wrong orders are informative too: a student who ranks moving trolleys by speed alone has used one reason where two were needed.

The limits: the items are on the screen, so nothing is recalled, and a right order does not prove a right reason. EdProcto's own note on the shape adds that a ranking without an exact criterion is an opinion.

Is forced ranking better than rating each item?

In survey research, forced ranking prevents the flat answer and costs the respondent more effort, and a quiz faces the same trade.

Krosnick and Alwin (1988) found that some respondents asked to rate valued qualities avoided hard choices by rating every quality as highly and equally desirable. With those respondents removed, ratings gave results like those rankings usually give, and the authors concluded that ranking may be the superior method for measuring values.

The cost is effort. In a practitioner study, Sauro et al. (2023) had 541 online panel members rank lists of 4 to 30 items. The first ranking question in a survey took about 7 seconds plus about 5 seconds for each item, and felt harder as the list grew.

These are surveys, where no answer is marked, but the mechanism carries over: a ranking will not let a student call everything about equal, and a long list tests patience.

How do you write a ranking question?

Write it around one criterion with a stated direction, on four to six items that differ enough to be told apart. We found no tested rules for the shape, so these steps follow from the evidence above.

1. State one criterion and its direction. "Rank from the largest kinetic energy to the smallest", never "rank by energy".
2. Make the items differ enough. Unless calculating is the point, the gaps should be visible without a calculator. In the first example below, each energy is at least a third larger than the next.
3. Settle ties before the quiz. A forced ranking cannot say "equal". Change a value until no two items are level, or ask about the tie separately. If you allow ties, say so in the stem and mark that question yourself: a key that holds one order cannot credit two.
4. Check that no single feature gives the key. If the right order is also the order by speed alone, the shortcut earns full marks.
5. Use four to six items. Four items can be arranged in 24 ways, five in 120 and six in 720, with 6, 10 and 15 pairs to judge, all computed in code. Three items can be guessed whole one time in six. Eight ask for 28 comparisons, in any of 40,320 orders, to earn one question's marks.
6. Build it. In EdProcto, choose Ranking and type the items in the right order, one a line.

Ranking question examples, marked for three students

Each ranking below carries 5 marks, and each student's order was run through EdProcto's own marking function.

Physics: five trolleys by kinetic energy

Five trolleys roll along a level track. Rank them from the largest kinetic energy to the smallest.

• P: 1 kg at 6 m/s
• Q: 2 kg at 2 m/s
• R: 4 kg at 5 m/s
• S: 8 kg at 3 m/s
• T: 12 kg at 1 m/s

Kinetic energy is half the mass times the square of the speed: R 50 J, S 36 J, P 18 J, T 6 J and Q 4 J. The key, RSPTQ, matches none of the orders by mass, speed or momentum.

Student 2 has eight of the ten pairs right and two reversed, a net six tenths of 5; student 3 has seven and three. Ranking by mass alone, TSRQP, nets nothing on either setting. Each wrong order points at a rule.

Likelihood: two dice in a board game

You roll two fair dice. Rank these results from most likely to least likely.

• At least one die shows a six
• Both dice show odd numbers
• The total is 7
• The total is 4
• The total is 12

Of the 36 equally likely rolls, 11 show at least one six, 9 have both dice odd, 6 total 7, 3 total 4 and 1 totals 12, counted in code. The key is the list as printed.

Students 1 and 2 each reversed one neighbouring pair and earn the same 4 marks, although student 2 named the wrong result as most likely. The pair rule gives the top place no extra weight.

How do you rank by likelihood for diagnosis and risk?

Rank by likelihood when the judgement is which explanation or outcome is most probable, and decide first whether the top place deserves more than its share of the marks. In diagnosis, fault-finding and risk assessment, the top of the list decides what is done next.

Semigran et al. (2015) put 45 standardised patient cases through 23 online symptom checkers and judged them by how often the correct diagnosis was listed first (34% of evaluations) and how often in the first 20 (58%). In information retrieval, Vigna (2015) notes that Kendall's tau, the pair-counting statistic most used to compare two rankings, fails in its standard form to capture that disagreements between high-ranked items matter more, and proposes a weighted form that also copes with ties.

EdProcto lists rank by likelihood with its selection shapes, but the student orders the options and the answer is marked exactly as a ranking is. To give the top its weight, ask "Which is most likely?" first as a multiple-choice question with its own points.

The key needs a source: frequencies you can count or probabilities you can compute, as with the dice. Where the likelihoods are professional judgement, agree the key with a panel, as for best and worst questions.

How should a ranking question be scored?

Score it all or nothing when only the complete order will do, by pairs when partial knowledge should earn marks, and by the top choice when first place is what gets acted on.

The guide to ordering questions compares six scoring methods on one question, so that is not repeated here. All or nothing gave five of the six students above no marks, though none was guessing: a blind guess at five items is right one time in 120.

Ask for the reasoning behind one adjacent pair

Explaining every placement is slow to write and to mark, so ask about one adjacent pair, the one where the right rule and the tempting rule disagree: "Take trolleys R and S. Which has more kinetic energy, and how do you know?"

EdProcto's two shapes that ask for a reason, choose and justify and sort and justify, attach it to a choice or a sort, not to a ranking. So follow the ranking with a short answer question. It waits on the grading page, where you mark it against your model answer or, from the Tutor plan up, press "Ask Procto AI" on that answer and keep or change the mark.

How does EdProcto build and mark a ranking question?

EdProcto is the best tool for a ranking question when the items come from your own course and the mark has to be explained to a student: the key is one order you typed, and the rule can be said in a sentence.

• Building. In the quiz maker on the home page, paste your notes and choose Ranking or Rank by likelihood, and Procto AI drafts questions for you to check. Or type the items yourself. Word each line differently, so no two items read the same.
• Answering. The student sees the items shuffled in a numbered list and moves each with up and down arrows; there is no dragging. A list left untouched counts as no answer until an item is moved or "keep this order" is pressed.
• Marking. By rule, at hand-in. "Partial credit" is a setting of the quiz, off by default. Off, only the exact order earns the marks. On, the mark is the pairs in the right relative order minus the reversed pairs, as a share of all pairs, never below zero. Nothing in the rule handles ties, and the top place carries no extra weight.
• On paper. The items print lettered and out of order, the same on every copy, and the key gives the order by those letters. See printing a quiz with an answer key.
• Results. For an ordering question the analysis reports the average share of the marks earned, how far the strongest students outscored the weakest, and the median time. It does not count which orders were given, so ask a contested pair as multiple choice.

The Free plan covers ten quizzes a month with the core templates and marking by rule, Tutor unlocks all 111 templates, and the full analysis comes with Educator. Prices are on the pricing page.

What are the common faults in ranking questions?

The common faults make the order arguable, guessable or impossible to mark.

• No direction. "Rank by density" leaves students guessing which end comes first.
• A criterion that is an opinion. "Rank these causes by importance" has no key a rule can hold.
• A hidden tie. Two items are level, and the key picks one order.
• Too many items. Past six, patience is what gets measured.
• An unstated rule. Students should know whether one swap costs a pair or the whole question.

Quick questions

How many items should a ranking question have?

Four to six. Four items have 24 possible orders and six have 720.

How do you score a ranking question?

All or nothing, or by pairs. EdProcto's ranking gives the marks for the exact order only, unless "Partial credit" is on, when each pair in the right relative order counts for the student and each reversed pair against.

What is the difference between a ranking question and an ordering question?

A ranking puts items along a scale, such as size or likelihood. An ordering question puts steps in the order they happen. EdProcto marks both with one rule.

Can two items tie in a ranking question?

Not in a forced ranking, which cannot say "equal", and EdProcto's key holds one order. Change a value until no two items are level, or ask about the tie separately.

References

• Hudgins, D. W., Prather, E. E., Grayson, D. J., & Smits, D. P. (2006). Effectiveness of collaborative ranking tasks on student understanding of key astronomy concepts. Astronomy Education Review, 5(1), 1–22. https://doi.org/10.3847/AER2006001
• Krosnick, J. A., & Alwin, D. F. (1988). A test of the form-resistant correlation hypothesis: Ratings, rankings, and the measurement of values. Public Opinion Quarterly, 52(4), 526–538. https://doi.org/10.1086/269128
• Maeyer, J., & Talanquer, V. (2010). The role of intuitive heuristics in students' thinking: Ranking chemical substances. Science Education, 94(6), 963–984. https://doi.org/10.1002/sce.20397
• Maloney, D. P. (1985). Rule-governed approaches to physics: Conservation of mechanical energy. Journal of Research in Science Teaching, 22(3), 261–278. https://doi.org/10.1002/tea.3660220308
• McClary, L., & Talanquer, V. (2011). Heuristic reasoning in chemistry: Making decisions about acid strength. International Journal of Science Education, 33(10), 1433–1454. https://doi.org/10.1080/09500693.2010.528463
• PhysPort. (n.d.). Ranking task exercises in physics. American Association of Physics Teachers. Retrieved October 10, 2026, from https://www.physport.org/methods/Ranking_Tasks
• Sauro, J., Atkins, D., Du, D., & Lewis, J. (2023, October 10). How hard is it to rank items in surveys? MeasuringU. https://measuringu.com/how-hard-is-it-to-rank-items-in-surveys/
• Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015). Evaluation of symptom checkers for self diagnosis and triage: Audit study. BMJ, 351, Article h3480. https://doi.org/10.1136/bmj.h3480
• Singh, C. (2008). Assessing student expertise in introductory physics with isomorphic problems. II. Effect of some potential factors on problem solving and transfer. Physical Review Special Topics - Physics Education Research, 4(1), Article 010105. https://doi.org/10.1103/PhysRevSTPER.4.010105
• Vigna, S. (2015). A weighted correlation index for rankings with ties. In Proceedings of the 24th International Conference on World Wide Web (pp. 1166–1176). International World Wide Web Conferences Steering Committee. https://doi.org/10.1145/2736277.2741088