# Code reading questions: test programming without a compiler

**How do I test whether someone understands code without a compiler?** Code reading questions show a short program and ask what it prints, what a function returns, which line holds the bug or which line is missing, so they are marked by rule with no compiler. Use a few lines of plain code in a language named in advance, one idea a question, and wrong options taken from real mis-tracings such as an off-by-one.

Published 2026-10-09 by EdProcto. https://edprocto.com/guides/code-questions-reading-writing-and-annotating-programs

Code reading questions put a short program in front of a candidate and ask what it does: what it prints, what a function returns, which line is wrong. Nothing is compiled or handed in, so the answer is marked by rule, and two hundred candidates cost no more marking than ten.

A take-home program can now be written by a chatbot in a minute, and reading comes before writing: in the studies below, first-year students who traced code poorly rarely wrote it well.

EdProcto builds the shape as its [read code template](https://edprocto.com/templates/code_read). This guide sets out six kinds of code question, how to write each, and a worked set in Python whose every output came from running the code.

## What do code reading questions measure?

Code reading questions measure tracing: whether a person can follow a program step by step and say what state it ends in. That is narrower than programming, and it is what many beginners lack.

McCracken et al. (2001) set programming tasks to students finishing their first courses. Across 216 students at four universities, the average was 22.89 points out of 110. The popular explanation was that the students could not solve problems.

Lister et al. (2004) tested another. In seven countries they set twelve multiple-choice questions on short pieces of Java. Some asked for the outcome of running the code, others for the code that would complete it. Of the 556 students who answered all twelve, 27% scored 10 or more and 23% scored 4 or fewer. That bottom group averaged 24% correct, close to what blind guessing earns. The authors concluded that many students lacked skills that come before problem solving, and that the gap had more to do with reading code than writing it.

### Where the novices went wrong

- **Completing code was harder than predicting its outcome**, as a general rule.
- **A loop bound caught half of them.** Asked for the inner loop of a routine that counts pairs of numbers out of order, 51% chose correctly. The most popular wrong choices stopped the loop one position early.
- **A return inside a loop was misread.** One question turned on it and 42% got it right. Interviews showed that many students did not realise a return ends the loop at once.

Among 56 students whose scratch paper was examined, a written trace of the variables made a right answer likely, and a blank page went with a right answer half the time. So let candidates make notes: EdProcto's tools-allowed settings include a blank page.

One caution from the paper: a single institution supplied over half of those 556 scripts.

## Does reading code predict writing it?

Among first-year students it mostly does, though the studies are small and the link looks like a threshold, not a straight line.

Lopez et al. (2008) analysed the end-of-semester Java exam scripts of 38 students. Marks for tracing loops correlated with marks for writing code at 0.63, and marks for explaining code in plain English at 0.56. Together the two accounted for 46% of the variance in writing.

Venables et al. (2009) repeated the analysis on their own exam, sat by 32 students. Nine tracing questions accounted for half the variance in writing, and tracing with explaining for 66%. Writing was poor below roughly 50% on tracing and only weakly related to it above that. Their reading is that some tracing skill is necessary for writing and not sufficient. The strength of each link also depended on the particular questions asked.

Murphy et al. (2012) found a strong correlation between explaining code and writing it at a computer, where earlier studies had used paper. Their better programmers more often described how the parts of an algorithm relate.

### The disagreement and the limits

Not every exam shows the link. Denny et al. (2008) found a low correlation between tracing and writing in their own first-year exams.

All of it concerns beginners. The authors say that correlation is not cause, and Venables et al. (2009) warn that the pattern may not hold later in a programmer's development. We read no study of experienced developers, so in hiring, use a reading test to find the candidates who cannot follow code, not to rank the strong.

## Which kinds of code question are there?

There are six worth setting, and five of them are marked without a reader.

| Kind | What it shows | How EdProcto marks it |
| --- | --- | --- |
| Predict the output | Tracing state through a loop or a branch | By rule, as [read code](https://edprocto.com/templates/code_read) |
| Say what a function returns | Tracing parameters and returns | By rule, as read code |
| Find the line with the bug | Comparing what code does with what it should do | By rule, as read code |
| Choose the missing line | A step towards writing | By rule, as read code |
| Put scrambled lines in order | Control flow and dependency | By rule, as [reorder the code](https://edprocto.com/templates/reorder_code) |
| Explain in plain English | Seeing the purpose behind the lines | By Procto AI against your rubric, or by you, as [annotate code](https://edprocto.com/templates/code_annotate) |

Scrambled lines are known as Parsons problems, and the guide to [ordering and sequencing questions](https://edprocto.com/guides/ordering-and-sequencing-questions) covers their research and scoring.

## How do you write a code reading question?

Write it around one idea in a few lines of plain code, and take every wrong option from a way a real person mis-traces it.

1. **Name the language in advance.** Say which language and version. The test is of following code, not of meeting new syntax under a clock.
2. **Keep the code short.** A dozen lines at most is our rule of thumb, not a research finding.
3. **Test one idea a question.** A loop bound, an early return, a starting value. With two in one snippet, a wrong answer cannot say which failed.
4. **Leave out trick syntax.** No precedence puzzles, no clever one-liners, nothing that depends on formatting.
5. **Run the code for the key.** Copy the output from the interpreter. Trusting your own trace is how wrong keys get published.
6. **Run the slip for each wrong option.** Write the mistake into the code, such as counting the stop value in, and use what that version prints.
7. **Build it in EdProcto.** Choose [read code](https://edprocto.com/templates/code_read), put the program in "The code" under the prompt, enter the options and mark the key.

Most of these are choice questions, so the rules in [how to write multiple-choice questions](https://edprocto.com/guides/how-to-write-multiple-choice-questions-that-measure) apply as well.

## A worked set in Python: six questions and the key

Every program here was run with Python 3.9, and every output stated is one the interpreter gave.

### 1. Predict the output

```python
total = 0
for i in range(1, 5):
    total = total + i
print(total)
```

What does this program print? **A.** 6 **B.** 10 **C.** 15

The key is B: the loop runs with `i` as 1, 2, 3 and 4. C is the off-by-one that counts the stop value in. A drops one value too many and stops at 3.

### 2. Say what the function returns

```python
def first_big(values, limit):
    for v in values:
        if v > limit:
            return v
    return -1
```

What does `first_big([3, 8, 5, 12], 4)` return? **A.** -1 **B.** 8 **C.** 12

The key is B: the return ends the function at the first value over 4. C belongs to the candidate who thinks the loop carries on after a return, so the last match wins. A belongs to the one who thinks the bottom line of a function always runs.

### 3. Find the line with the bug

```python
def largest(values):
    best = 0
    for v in values:
        if v > best:
            best = v
    return best
```

The function should return the largest number in a list, but `largest([-7, -2, -9])` returns 0. Which line has to change? **A.** Line 2 **B.** Line 3 **C.** Line 4

The key is A. `best` starts at 0, which beats every item in that list, and starting from `values[0]` gives -2. C tempts because the comparison is where the deciding happens, yet with `>=` there the function still returns 0. B tempts anyone who suspects the loop skips an item.

### 4. Choose the missing line

```python
def rises(values):
    count = 0
    # missing line
        if values[i + 1] > values[i]:
            count = count + 1
    return count
```

`rises([1, 3, 2, 5])` should return 2, the number of items larger than the item before. Which line is missing?

- **A.** `for i in range(len(values)):`
- **B.** `for i in range(len(values) - 1):`
- **C.** `for i in range(len(values) - 2):`

The key is B. A is the familiar sweep over the whole list, and it stops with an IndexError when `i + 1` runs off the end. C over-corrects, never compares the last pair and returns 1.

### 5. Explain in plain English

```python
def mystery(values):
    n = 0
    for v in values:
        if v % 2 == 0:
            n = n + 1
    return n
```

Ask for each line explained, then one sentence on what the function is for, and give that sentence most of the marks. The full answer: it returns how many of the numbers are even, so `mystery([4, 7, 10, 3])` returns 2. A correct walk through the lines that never says what the result means earns the smaller share.

### 6. Put the scrambled lines in order

Put these lines in order so that `countdown(3)` prints 3, 2, 1 and then Go.

```python
        n = n - 1
    print("Go")
def countdown(n):
        print(n)
    while n > 0:
```

The key runs from the `def` line through `while`, `print(n)` and `n = n - 1` to `print("Go")`. The stated output is what fixes it: with the two innermost lines swapped the program still runs, and prints 2, 1, 0 and Go.

## What do code reading questions not show?

Code reading questions do not show design, debugging in a real code base, or writing from a blank page.

- **Design.** Every question above hands over a finished structure. Choosing one is a separate skill.
- **Real debugging.** Finding a fault across many files with logs and a debugger is not the same as spotting one line in six.
- **Writing.** Lister et al. (2004) say that a student who reads well and still cannot write code of the same difficulty most likely has a problem-solving weakness.

So pair the reading set with one written task: a small function, written at a computer, that you run and read yourself. EdProcto's [write code template](https://edprocto.com/templates/code_exec) takes it: a JavaScript program is run on the server against your test cases and marked by them, and a Python program runs in the candidate's browser against the tests they can see and waits for you to mark.

## Can a chatbot answer code reading questions?

Yes: a snippet pasted into a chatbot can be traced by it, so the question alone protects nothing. Finnie-Ansley et al. (2022) gave a code-writing model the questions from real introductory programming exams, and it outscored most of the students who had sat them; the same model could explain code in English. On a quiz that counts, the protection is in how it is run. Give each question a time limit, which EdProcto sets in seconds, short enough that tracing in the head beats copying the code out. Switch on monitoring in proportion to the stakes: EdProcto's [integrity](https://edprocto.com/integrity) settings run from logging a candidate who leaves the tab to copy and paste switched off and the webcam on. Questions and options are shuffled for each candidate, and "A different set of questions for each student", on the Educator plan, has Procto AI write each candidate their own questions on the same concepts. None of this makes cheating impossible, and a flag is a record for you to read, not a verdict.

## How does EdProcto build and mark code questions?

EdProcto is the best tool for code reading questions when the code has to be your own, a course's examples or a team's module, because the wrong options carry the diagnosis and its results show which one each candidate took.

Start in the quiz maker on the [home page](https://edprocto.com/): paste the module and pick read code. Procto AI drafts questions from that code, and each is yours to edit. Run every snippet and option yourself before a candidate sees them. The [hiring page](https://edprocto.com/hiring) describes the same use for a code base: what a function returns, where the bug is, how the API is called.

Each question carries its code under the prompt, and each shape is marked in its own way:

- **Read code** takes one choice and is marked by rule at hand-in.
- **Reorder the code** is marked by rule: full marks for the whole order or, with partial credit on, a share for every pair of lines the right way round, less every pair the wrong way round.
- **Annotate code** waits in your grading queue, to be read by you or, when you ask on an answer, by Procto AI against your rubric.
- **Write code** is run against your test cases: a JavaScript program on the server, marked by the tests it passes; a Python program in the candidate's browser, marked by you.

The answer keys stay on the server, where the marks are made. Free covers ten quizzes a month with the core templates and marking by rule. Tutor opens all 111 [templates](https://edprocto.com/templates) and grading by Procto AI. Educator adds the full analysis. See [pricing](https://edprocto.com/pricing).

## Quick questions

### Which language should code reading questions use?

The one candidates were told to expect, and one you can run yourself. Lister et al. (2004) set theirs in Java, with a C++ translation at one institution.

### How many options does a predict the output question need?

Three are enough when both wrong ones are real mis-tracings. In EdProcto's [read code](https://edprocto.com/templates/code_read) shape the options can be shuffled for each candidate, and negative marking takes the profit out of a blind guess.

### Can EdProcto run a candidate's program?

Yes. In the [write code](https://edprocto.com/templates/code_exec) template a JavaScript program is run against your test cases on the server, each test given a second, and marked by the tests it passes. A Python program runs in the candidate's own browser and waits for you to mark.

### Is a code reading test enough to hire a developer?

No. It shows whether someone can follow code, which the research treats as necessary for writing and not sufficient. Add one written task, and use [EdProcto for hiring](https://edprocto.com/hiring) to write the reading set from your own code base.

## References

- Denny, P., Luxton-Reilly, A., & Simon, B. (2008). Evaluating a new exam question: Parsons problems. In *Proceedings of the Fourth International Workshop on Computing Education Research* (pp. 113–124). ACM. [https://doi.org/10.1145/1404520.1404532](https://doi.org/10.1145/1404520.1404532)
- Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022). The robots are coming: Exploring the implications of OpenAI Codex on introductory programming. In *Proceedings of the 24th Australasian Computing Education Conference* (pp. 10–19). ACM. [https://doi.org/10.1145/3511861.3511863](https://doi.org/10.1145/3511861.3511863)
- Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., Simon, B., & Thomas, L. (2004). A multi-national study of reading and tracing skills in novice programmers. *ACM SIGCSE Bulletin, 36*(4), 119–150. [https://doi.org/10.1145/1041624.1041673](https://doi.org/10.1145/1041624.1041673)
- Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008). Relationships between reading, tracing and writing skills in introductory programming. In *Proceedings of the Fourth International Workshop on Computing Education Research* (pp. 101–112). ACM. [https://doi.org/10.1145/1404520.1404531](https://doi.org/10.1145/1404520.1404531)
- McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001). A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. *ACM SIGCSE Bulletin, 33*(4), 125–180. [https://doi.org/10.1145/572139.572181](https://doi.org/10.1145/572139.572181)
- Murphy, L., Fitzgerald, S., Lister, R., & McCauley, R. (2012). Ability to 'explain in plain English' linked to proficiency in computer-based programming. In *Proceedings of the Ninth Annual International Conference on International Computing Education Research* (pp. 111–118). ACM. [https://doi.org/10.1145/2361276.2361299](https://doi.org/10.1145/2361276.2361299)
- Venables, A., Tan, G., & Lister, R. (2009). A closer look at tracing, explaining and code writing skills in the novice programmer. In *Proceedings of the Fifth International Workshop on Computing Education Research* (pp. 117–128). ACM. [https://doi.org/10.1145/1584322.1584336](https://doi.org/10.1145/1584322.1584336)