# Ordering and sequencing questions: how to write and score them

**How do I write and score ordering questions?** An ordering question gives students the shuffled steps of a process and asks for the right sequence. Use four to seven steps, one action each, with a single defensible order and no wording that gives a position away. Choose the scoring rule before the quiz, because all or nothing, exact position and relative order can mark the same answer very differently.

Published 2026-10-09 by EdProcto. https://edprocto.com/guides/ordering-and-sequencing-questions

An ordering question hands the student the steps of a process in a shuffled list and asks for them back in sequence: the stages of mitosis, the steps of a titration, the lines of a proof. It tests the structure of a whole process, which no single multiple-choice question can.

It is also a shape where the scoring rule can decide the mark. An answer with one misplaced step can earn nothing under one common rule and four fifths of the marks under another.

EdProcto builds the shape as its [sequence the process template](https://edprocto.com/templates/process_seq), shuffles the steps and marks the answer by rule.

## What does an ordering question measure?

An ordering question measures whether a student knows what has to come before what. "Which stage of mitosis comes first?" checks one link in a chain. Six steps in order hold fifteen before-and-after relations, one for every pair of steps, and one answer commits the student to all of them.

The best-studied version comes from programming. A Parsons problem gives students the jumbled blocks of a working program to put back in their original order (Du et al., 2020). Denny et al. (2008) tried it in the exams of a first programming course: marks on it tracked marks for writing code, and neither was closely related to marks for tracing code. In mathematics, Poulsen et al. (2021) set proofs as prewritten lines to be dragged into order, in the exams of a course with hundreds of students. They were easier than written proofs and gave about as much information about what students knew.

Two cautions. Reviewing the Parsons research, Du et al. (2020) concluded that how well the format works as a question type remains uncertain, because so little of the work has been replicated. And nearly all of it concerns code: nothing read for this guide tests the ordering of a laboratory procedure or a first-aid sequence, so the advice for those rests on the logic of the shape, not on experiments.

The shape also leaves recall out: the steps are on the screen, and the student only arranges them. Where recall is the point, EdProcto's [list in order](https://edprocto.com/templates/list_ordered) has the student type the steps.

## How do you write an ordering question?

Write it from a process with exactly one defensible order, in four to seven short steps, and check that no step announces its own place. We found no tested rulebook for this shape, so the rules below apply the logic of [multiple-choice item writing](https://edprocto.com/guides/how-to-write-multiple-choice-questions-that-measure): one task, parts alike, no clue in the wording.

1. **Choose a process with one defensible order.** For each pair of neighbouring steps, ask whether a competent person could do them the other way round. If so, merge them, drop one, or settle it in the stem.
2. **Name the start of a cycle.** The cell cycle has no first stage until the stem gives one.
3. **Use four to seven steps.** Four steps can be arranged in 24 ways and seven in 5,040. Fewer invites guessing, and more tests patience with a list.
4. **Give each step one action.** "Fill the burette and record the level" is two steps.
5. **Remove the position words:** first, then, next, finally, again, and any "it" that points back at an earlier step.
6. **Keep the steps alike** in length and grammar.
7. **Be wary of decoy steps.** Parsons problems often carry extra lines, called distractors, that belong in no solution. Smith et al. (2023) compared 32 pairs of such problems, with and without the extras, in programming exams and quizzes: the extras cost students a lot of time and did little for how well the questions separated them.
8. **Build it and set the rule.** In EdProcto, enter the steps in their right order under [sequence the process](https://edprocto.com/templates/process_seq), and decide on partial credit before anyone sits the quiz.

### A sequence, before and after

Before:

> Put the steps of a titration in order. First, rinse the burette. Then fill it and take a reading. Next, add it to the flask. Finally, it changes colour and you take the final reading.

Every step wears a signpost, two steps hold two actions, and "it" is the burette, then the acid, then the indicator.

After:

> A flask holds 25 cm³ of sodium hydroxide solution and three drops of indicator. Put the steps for titrating it with hydrochloric acid in order.
>
> - **A.** Rinse the burette with the acid.
> - **B.** Fill the burette with the acid.
> - **C.** Record the level of the acid.
> - **D.** Run acid into the flask while swirling.
> - **E.** Stop when one drop changes the colour.
> - **F.** Work out the volume of acid used.

The letters give the right order here; students see the steps shuffled, without them. The stem starts with the flask ready, which settles the steps that could float, and only one order works.

## How should you score an ordering question?

Score it by the rule that fits what the sequence is for, and pick that rule before the quiz, because the methods can disagree by most of the marks on one answer. Six are worth knowing.

| Method | How the mark is made | Rewards | Punishes |
| --- | --- | --- | --- |
| All or nothing | Full marks for the whole order, else zero. EdProcto's rule with partial credit off | Complete command | Any slip at all |
| Exact position | A share for each step in its own place | Knowing where each step sits | An early slip, which shifts every later step |
| Adjacent pairs | A share for each step directly followed by its true successor | Knowing what comes next | A swap of neighbours, which can break three links |
| Relative order | A share for every pair of steps the right way round | The overall direction | Little: a random answer earns about half |
| Relative order, less reversals | Pairs the right way round, less pairs the wrong way round, over all pairs, never below zero. The rule [sequence the process](https://edprocto.com/templates/process_seq) uses with partial credit on | The overall order, however far the slip | A step moved far more than a step one place out; a backwards order earns nothing |
| Edit distance | Marks fall with each step taken out and put back | Long runs in the right order | Every moved step alike, near or far |

Relative order is the count behind Kendall's tau: the number of pairs the wrong way round is the Kendall tau distance, which equals the number of swaps of neighbours needed to repair the answer. The total distance of the steps from their true places always lies between that count and twice it (Diaconis and Graham, 1977). Edit distance is how one proof-ordering tool gives partial credit: the number of lines, less the removals and insertions needed to reach a correct proof, over the number of lines (Poulsen et al., 2023).

### One slip, scored six ways

Take the titration question at 6 marks. Student 1 puts "Rinse the burette" last, as if it were washing up, and has everything else right: B C D E F A. Student 2 swaps two neighbours, recording the level after the acid has started to run: A B D C E F.

| Method | Student 1 | Student 2 |
| --- | --- | --- |
| All or nothing | 0 | 0 |
| Exact position | 0 | 4 |
| Adjacent pairs | 4.8 | 2.4 |
| Relative order | 4 | 5.6 |
| Relative order, less reversals | 2 | 5.2 |
| Edit distance | 4 | 4 |

The working:

- **Exact position.** None of student 1's steps is in its own place. Student 2 has four of six in place.
- **Adjacent pairs.** Of the five links, A to B through E to F, student 1 keeps four (four fifths of 6 is 4.8) and student 2 keeps two (2.4).
- **Relative order.** Six steps make 15 pairs. Student 1 has the five pairs involving A the wrong way round, leaving 10 of 15. Student 2 has one pair wrong, leaving 14.
- **Relative order, less reversals.** Student 1 has 10 pairs right and 5 wrong, a net 5 of 15, which is a third of 6. Student 2 has 14 right and 1 wrong, a net 13 of 15.
- **Edit distance.** Each answer is one removal and one insertion from the key: six less two, over six.

The same two answers can be ranked either way round. Exact position gives student 1 nothing and student 2 two thirds, while adjacent pairs gives student 1 twice what it gives student 2. Neither is an error. One rule asks whether each step is where it should be, the other whether the student knows what follows what.

### What the research says about the choice

The research does not settle it. Eggen and Lampe (2011) compared scoring methods for multiple-response, matching and sequencing questions by working out, from assumed levels of knowledge, how reliable each method would be on a single question. For matching they could recommend simple partial credit, a count of the right pairs. For sequencing, the results were not as conclusive as for the other two types.

The proof-ordering paper shows that edit-distance credit can be computed fast enough for a large class, not that its marks are fairer (Poulsen et al., 2023). We found no study that marked one set of real ordering answers by several methods and checked each against another measure of what the students knew.

Until one exists, choose by purpose: all or nothing where any wrong order is a failure, partial credit for a lesson check, and tell the class the rule beforehand.

## Can a student guess an ordering question?

Hardly at all for full marks, and only a little under partial credit. A blind guess at n steps is fully right once in n factorial tries, and under exact-position credit it earns one step's worth on average, whatever the length.

| Steps | Fully right by guessing | Average guess, exact position |
| --- | --- | --- |
| 4 | 1 in 24 | 25% |
| 5 | 1 in 120 | 20% |
| 6 | 1 in 720 | 17% |
| 7 | 1 in 5,040 | 14% |

The reason is short: each step has a 1 in n chance of landing in its own place and there are n steps, so a random order has one step right on average. For six steps, 265 of the 720 orders (37%) put no step where it belongs.

Adjacent pairs has the same floor, since each link survives a shuffle with a chance of 1 in n. Relative order does not: a random answer has half its pairs the right way round, so it needs rescaling first. Taking the reversed pairs away does that. Counted over every possible order for this guide, a blind guess then averages 19% for four steps, 16% for five, 15% for six and 13% for seven, and half or more of all orders earn nothing.

## How does EdProcto build and mark an ordering question?

EdProcto is the best tool for an ordering question when the process comes from your own protocol or handout and the mark has to stand up. It drafts the steps from that text, keeps the key on the server, and gives partial credit by relative order less reversals, where one misplaced step costs what its distance deserves and a blind guess averages about a sixth of the marks.

Paste the protocol into the quiz maker on EdProcto's [home page](https://edprocto.com/) and choose sequence the process. Procto AI drafts the steps for you to edit, or you type them yourself in the right order. The template holds one right order and only the steps that belong to it: no decoy steps, and a process with two valid orders has to be reworded until it has one.

The student sees the steps shuffled, using a seed that the student never receives, so the page holds no answer key. Marking is by rule and immediate. The partial credit setting chooses the method, and it is off until you switch it on:

- **Off: all or nothing.** Full points when every step is in its place, otherwise zero.
- **On: relative order, less reversals.** EdProcto looks at every pair of steps. A pair the right way round counts for the student and a pair the wrong way round counts against. The net count, over the number of pairs, is the share of the question's points, and it never falls below zero. A step left out counts neither way.

In the worked example, student 2 scores 5.2 of 6 with partial credit on and nothing with it off, and student 1 scores 2 with it on and nothing with it off. The rule is strict about distance, not about position: a step one place out costs a single pair, a step carried from the start to the end costs every pair it crossed, and an order written backwards earns nothing.

The same rule marks [reorder the code](https://edprocto.com/templates/reorder_code), which is a Parsons problem, the [timeline](https://edprocto.com/templates/timeline) and the [ranking](https://edprocto.com/templates/ranking). Free covers ten quizzes a month with the core templates and marking by rule. Tutor opens all 111 [templates](https://edprocto.com/templates), and Educator adds the full analysis of how hard each question was and how well it separated students. See [pricing](https://edprocto.com/pricing).

## What are the common mistakes?

The common mistakes make the order easier to find than the process is to know, or hide the marking rule.

- **Two defensible orders.** A student who chose the other valid order is marked wrong for being right.
- **Obvious ends.** If "collect the equipment" opens the list and "write up the results" closes it, six steps are really four.
- **An unstated rule.** Students told only that partial credit applies cannot know whether one early slip costs a mark or the whole question.

## When should you use matching, a timeline or a ranking instead?

Use another shape when the knowledge is not a fixed chain of steps.

- **Pairs, not a chain.** Terms against definitions, or causes against effects, belong in a [matching question](https://edprocto.com/guides/matching-questions-how-to-write-them-well).
- **Dates, not dependencies.** Events ordered only by when they happened go on a [timeline](https://edprocto.com/templates/timeline).
- **A scale, not a sequence.** Items ordered by size, reactivity or likelihood under a stated criterion are a [ranking](https://edprocto.com/templates/ranking).
- **One link only.** If a single step is what matters, ask it as a [multiple-choice question](https://edprocto.com/guides/how-to-write-multiple-choice-questions-that-measure).

## Quick questions

### How many steps should an ordering question have?

Four to seven. Four steps have only 24 possible orders, and past seven the list is hard to hold in view. EdProcto's note on [sequence the process](https://edprocto.com/templates/process_seq) gives the same range.

### Should students get partial credit on ordering questions?

In most classroom quizzes, yes, with the rule explained first. Keep all or nothing where any wrong order is a failure. The one comparison of scoring methods read for this guide was inconclusive for sequencing (Eggen & Lampe, 2011).

### What is the Kendall tau distance?

The Kendall tau distance is the number of pairs of steps a student has the wrong way round, which equals the neighbour swaps needed to repair the answer. For six steps it runs from 0 to 15.

### Why did a student with one step wrong score zero?

Under all or nothing, any slip is zero, and under exact-position credit a step moved from the start to the end shifts every other step by one place, so none sits where the key has it. EdProcto's partial credit avoids the second case: it marks the order of the steps against each other, so the rest of the sequence still earns its share.

## References

- Denny, P., Luxton-Reilly, A., & Simon, B. (2008). Evaluating a new exam question: Parsons problems. In *Proceedings of the Fourth International Workshop on Computing Education Research* (pp. 113–124). ACM. [https://doi.org/10.1145/1404520.1404532](https://doi.org/10.1145/1404520.1404532)
- Diaconis, P., & Graham, R. L. (1977). Spearman's footrule as a measure of disarray. *Journal of the Royal Statistical Society: Series B (Methodological), 39*(2), 262–268. [https://doi.org/10.1111/j.2517-6161.1977.tb01624.x](https://doi.org/10.1111/j.2517-6161.1977.tb01624.x)
- Du, Y., Luxton-Reilly, A., & Denny, P. (2020). A review of research on Parsons problems. In *Proceedings of the Twenty-Second Australasian Computing Education Conference* (pp. 195–202). ACM. [https://doi.org/10.1145/3373165.3373187](https://doi.org/10.1145/3373165.3373187)
- Eggen, T. J. H. M., & Lampe, T. T. M. (2011). Comparison of the reliability of scoring methods of multiple-response items, matching items, and sequencing items. *CADMO, 19*(2), 85–104. [https://doi.org/10.3280/CAD2011-002008](https://doi.org/10.3280/CAD2011-002008)
- Poulsen, S., Kulkarni, S., Herman, G., & West, M. (2023). Efficient feedback and partial credit grading for Proof Blocks problems. In *Artificial intelligence in education* (pp. 502–514). Springer. [https://doi.org/10.1007/978-3-031-36272-9_41](https://doi.org/10.1007/978-3-031-36272-9_41)
- Poulsen, S., Viswanathan, M., Herman, G. L., & West, M. (2021). Evaluating Proof Blocks problems as exam questions. In *Proceedings of the 17th ACM Conference on International Computing Education Research* (pp. 157–168). ACM. [https://doi.org/10.1145/3446871.3469741](https://doi.org/10.1145/3446871.3469741)
- Smith, D. H., Fowler, M., & Zilles, C. (2023). Investigating the role and impact of distractors on Parsons problems in CS1 assessments. In *Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1* (pp. 417–423). ACM. [https://doi.org/10.1145/3587102.3588819](https://doi.org/10.1145/3587102.3588819)