Parsons problems: what they are, examples and how to score them

A Parsons problem hands a student every line of a working program in scrambled order and asks for the program back. Nothing is typed, so no mark is lost to a missing colon, and the answer can be marked by rule.

The format is twenty years old and, as practice, holds up well against writing code. What it cannot show is that the student could have written the program alone.

EdProcto builds the shape as its reorder the code template. This guide covers the research, two worked examples in Python, and one wrong answer scored five ways.

What is a Parsons problem?

A Parsons problem is a programming exercise in which the lines of a correct program are given out of order and the student arranges them into a program that works.

Dale Parsons and Patricia Haden described it in 2006 as Parson's Programming Puzzles. Mastering basic syntax and logic is essential, they wrote, and the drill it takes can be tedious. Their answer was an automated puzzle whose pieces a tutor could design to bring out one topic or one common error, and whose solution was always a whole piece of sound code (Parsons & Haden, 2006).

It now takes several forms.

Are Parsons problems effective?

For practice, yes on the evidence so far: students finish sooner and, in most comparisons, learn about as much as by writing the code.

Against writing the same code

• Ericson et al. (2017) set two-dimensional problems with distractors against fixing code with the same errors and writing the equivalent code. The Parsons group took significantly less time, with no significant difference in learning or in retention a week later.
• Ericson et al. (2018) found adaptive problems more efficient practice than writing the code, and as effective.
• Weinman et al. (2021) tried faded problems with 237 students. For teaching programming patterns they beat writing and tracing exercises, and they improved code writing about as much as writing exercises did.

As an exam question

Denny et al. (2008) tried Parsons problems in a set of first-course programming exams. The marks correlated notably with marks for writing code and only weakly with marks for tracing it.

Distractor lines

Here the findings split.

• Harms et al. (2016), teaching young novices, found that distractors made no difference on a later transfer task, raised cognitive load, cut success at completing the problems by 26% and added 14% to the time.
• Smith et al. (2023) compared 32 pairs of problems from Python exams and quizzes. Distractors cost much more time and did little for discrimination, already high without them.
• Ericson et al. (2023) report evidence from think-aloud sessions and experiments that distractors help students learn to recognise, fix and avoid common errors.

Our reading: a distractor can earn its place in practice aimed at one error, and is hard to defend in a timed exam.

What the reviews say

Du et al. (2020) reviewed the work in introductory courses and would not call the format proven, as a question type or as a learning tool, because too few results had been repeated. Ericson et al. (2023) note that most earlier studies were each run at a single institution. None of the studies read for this guide reports testing experienced programmers.

How do you write a Parsons problem?

Write it from a short program that runs, make one order the only one that works, and decide about indentation and distractors on purpose. We found no tested rulebook; these steps follow the findings above and the logic of the shape.

1. Start from a program that runs. Put a call and its result in the prompt.
2. Keep it to five to eight lines. Our rule of thumb, not a finding: five lines have 120 possible orders and eight have 40,320.
3. Make one order the only one that works. Look for lines that could swap: two starting values, two independent updates, two prints. Merge them or rewrite the program, as in example 2.
4. Make every line different. Two copies of a line cannot be told apart.
5. Use the solution a student would expect. Haynes and Ericson (2021) found students usually more efficient on a Parsons problem than on writing the same code, but not when its solution was unusual.
6. Decide on indentation. Give it, and only the order is tested. Ask for it, and the problem is two-dimensional. Leave it out, and check that one order works whatever the indentation.
7. Leave distractors out of an exam. In practice, add one only when the error is the point, and take it from a real slip, as in how to write distractors.
8. Build it and fix the scoring rule. In EdProcto, choose reorder the code, paste the lines in working order, and decide on partial credit before anyone sits the quiz.

What does a Parsons problem look like? Two examples in Python

A Parsons problem looks like a short program cut into lines and shuffled, under a prompt that says what the finished program must do. Both programs below were run with Python 3.9, and a script tried every order of the lines, under every indentation Python accepts, against three test calls.

Example 1: five lines, indentation given

Put the lines in order so that initials("ada lovelace") returns "AL".

        out = out + word[0].upper()
    return out
def initials(name):
    for word in name.split():
    out = ""

The key:

def initials(name):
    out = ""
    for word in name.split():
        out = out + word[0].upper()
    return out

Of the 120 orders, one passes.

Example 2: four right answers cut to one

A first draft averaged the marks of 50 and over with a running total and a count: eight lines, among them two starting values and two updates. Four orders of those lines pass the tests, because each pair can swap. A key holds one order, so three correct answers would be marked wrong. Collecting the passes in a list removes both pairs.

Put the lines in order so that mean_of_passes([70, 40, 90]) returns 80.0. Indentation is left out.

passes.append(m)
return sum(passes) / len(passes)
if m >= 50:
def mean_of_passes(marks):
passes = []
for m in marks:

The key:

def mean_of_passes(marks):
    passes = []
    for m in marks:
        if m >= 50:
            passes.append(m)
    return sum(passes) / len(passes)

Of the 720 orders, one passes, whatever the indentation.

How should you score a Parsons problem?

Decide before the quiz whether a nearly right order earns anything, because the usual rules give the same answer very different marks.

Take example 2 at 6 points. A student makes the empty list just before using it, which puts the key's lines in the order 1, 3, 4, 2, 5, 6. With both lines under the if, that program runs and returns 90.0: each pass wipes the list.

Every figure came from a script, and the two EdProcto rows from its own scoring code, run on this answer.

One slip earns four different part marks. By position charges three lines for it, because moving a line down pushed two others up. The longest common subsequence counts the lines that would have to be lifted out and put back, here one, so a line moved one place costs the same as a line moved five. The pair counts fall with distance.

PrairieLearn (n.d.), in documentation read on 9 October 2026, offers partial credit by longest common subsequence, accepts every valid order once the author declares which blocks depend on which, and has settings for indentation and distractor blocks. EdProcto's template has none of those three.

Over all 720 orders of six lines, a blind guess averages 52% under the longest common subsequence, 50% under plain pairs, 17% by position and 15% under pairs right less pairs wrong. The guide to ordering and sequencing questions works through more methods.

How does EdProcto build and mark a Parsons problem?

EdProcto is the best tool for a Parsons problem when the lines must come from your own course code and the mark must be made out of students' reach. It drafts from your material, keeps the right order on the server, and gives partial credit by the rule under which a blind guess at six lines averages 15%.

• Writing. In the quiz maker on the home page, paste your code and pick reorder the code: Procto AI drafts the question for you to edit and run. Or paste the lines yourself, in the right order, one a line.
• What is kept. One accepted order. EdProcto strips the spaces from both ends of every line and drops blank lines, so indentation is not kept. There is no indentation step and no way to mark a line as a distractor.
• Answering. Each student sees the lines shuffled, as numbered rows with an up and a down arrow. A press swaps a line with its neighbour; there is no dragging. A student who finds the shuffle already right must press "keep this order", because an untouched list counts as no answer.
• Marking. By rule, when an ordinary quiz is handed in. With the grading setting "Partial credit" off, as on a new quiz, only the exact order earns the points. With it on, each pair of lines the right way round adds one and each pair the wrong way round takes one away; the total over the number of pairs, never below zero, is the share of the points. The switch covers the whole quiz, so it also changes multiple-answer, matching and sorting questions.

Working within those limits

• Write for order alone, as both examples are, and say in the prompt that indentation is left out.
• For spotting an error, the job a distractor does, ask which line holds the bug in a read code question.
• Keep it short: eight lines can take 28 presses of the arrows.
• Check the key before the quiz opens. A key changed afterwards reaches the attempts already handed in when you press "Re-mark this quiz" on the results page.
• On paper the lines print lettered and out of order, the same on every copy, and without their indentation, so a screen is still the better place for this shape.

On Free you get ten quizzes a month, the core templates and marking by rule. Tutor brings all 111 templates, and Educator the full analysis of each question. See pricing.

What can a Parsons problem not show?

A Parsons problem cannot show that the student could write the program unaided. Every line is supplied: the names, the syntax, the choice of loop, the plan. Arranging them takes reading and reasoning about order, while writing starts with recall and design. The correlation with writing marks in Denny et al. (2008) comes from one set of exams and is no licence to drop the written question.

Nor is the shape safe from a chatbot: Hou et al. (2024) showed a large multimodal model images of Parsons problems, and it solved 96.7% of them. On a quiz that counts, set a time limit and the integrity measures the stakes need. None makes cheating impossible, and a flag is a record for you to read, not proof.

So pair the reordering with one small program written from a blank page. EdProcto's write code template runs a JavaScript program against your test cases and marks it by them; a Python program runs in the student's browser and waits for you to mark. The guide to code reading questions covers the shapes in between.

Quick questions

What is a Parsons problem in simple terms?

A programming question in which the lines of a working program are shuffled and the student puts them back in order. EdProcto's version is reorder the code.

Are Parsons problems easier than writing code?

Typically easier and faster (Ericson et al., 2023). In Ericson et al. (2017) they took significantly less time than writing the equivalent code, and learning did not differ significantly.

Should a Parsons problem have distractors?

Not in a timed exam: Smith et al. (2023) found they cost time and added little. In practice they may help students learn to spot errors (Ericson et al., 2023).

Can I print a Parsons problem from EdProcto?

Yes. The paper prints the lines lettered and out of order, the same on every copy, and its key gives the order by those letters. On screens each student's lines are shuffled separately, which a single printed order cannot match.

References

• Denny, P., Luxton-Reilly, A., & Simon, B. (2008). Evaluating a new exam question: Parsons problems. In Proceedings of the Fourth International Workshop on Computing Education Research (pp. 113–124). ACM. https://doi.org/10.1145/1404520.1404532
• Du, Y., Luxton-Reilly, A., & Denny, P. (2020). A review of research on Parsons problems. In Proceedings of the Twenty-Second Australasian Computing Education Conference (pp. 195–202). ACM. https://doi.org/10.1145/3373165.3373187
• Ericson, B. J., Foley, J. D., & Rick, J. (2018). Evaluating the efficiency and effectiveness of adaptive Parsons problems. In Proceedings of the 2018 ACM Conference on International Computing Education Research (pp. 60–68). ACM. https://doi.org/10.1145/3230977.3231000
• Ericson, B. J., Margulieux, L. E., & Rick, J. (2017). Solving Parsons problems versus fixing and writing code. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research (pp. 20–29). ACM. https://doi.org/10.1145/3141880.3141895
• Ericson, B. J., Pearce, J. L., Rodger, S. H., Csizmadia, A., Garcia, R., Gutierrez, F. J., Liaskos, K., Padiyath, A., Scott, M. J., Smith, D. H., Warriem, J. M., & Zavaleta Bernuy, A. (2023). Multi-institutional multi-national studies of Parsons problems. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (pp. 57–107). ACM. https://doi.org/10.1145/3623762.3633498
• Harms, K. J., Chen, J., & Kelleher, C. L. (2016). Distractors in Parsons problems decrease learning efficiency for young novice programmers. In Proceedings of the 2016 ACM Conference on International Computing Education Research (pp. 241–250). ACM. https://doi.org/10.1145/2960310.2960314
• Haynes, C. C., & Ericson, B. J. (2021). Problem-solving efficiency and cognitive load for adaptive Parsons problems vs. writing the equivalent code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3411764.3445292
• Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024). More robots are coming: Large multimodal models (ChatGPT) can solve visually diverse images of Parsons problems. In Proceedings of the 26th Australasian Computing Education Conference (pp. 29–38). ACM. https://doi.org/10.1145/3636243.3636247
• Ihantola, P., & Karavirta, V. (2011). Two-dimensional Parson's puzzles: The concept, tools, and first observations. Journal of Information Technology Education: Innovations in Practice, 10, 119–132. https://doi.org/10.28945/1394
• Parsons, D., & Haden, P. (2006). Parson's programming puzzles: A fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education, Volume 52 (pp. 157–163). https://dl.acm.org/doi/10.5555/1151869.1151890
• PrairieLearn. (n.d.). pl-order-blocks element. Retrieved October 9, 2026, from https://docs.prairielearn.com/en/latest/elements/pl-order-blocks/
• Smith, D. H., Fowler, M., & Zilles, C. (2023). Investigating the role and impact of distractors on Parsons problems in CS1 assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (pp. 417–423). ACM. https://doi.org/10.1145/3587102.3588819
• Weinman, N., Fox, A., & Hearst, M. A. (2021). Improving instruction of programming patterns with faded Parsons problems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3411764.3445228