AI grading of open answers: how to audit the marks

A machine reader has marked every written answer and given each a tidy reason. You glance down the column and it looks right. That glance is not an audit.

An audit of AI grading of open answers is a small measurement, made before anyone relies on the marks.

EdProcto, which publishes this guide, has Procto AI propose a mark on one written answer at a time and records who gave every mark. Which answers to read first is in our guide to checking the marks it proposes; the studies here are ones it does not use, each read at source on 10 October 2026, as an abstract where we say so.

What is an audit of machine marking, and why is "it looks right" not one?

An audit is a measurement: a person marks a sample again, blind, and the two sets of marks are compared.

• A sample drawn by chance, so nobody picks the answers that will agree.
• Marks made blind, before the marker has seen the machine's.
• Two figures. Exact agreement is the share of sampled answers with identical marks; agreement within one point also counts those a point apart.
• Every disagreement read, because only the answers say why.

"It looks right" fails on each count. The eye falls where it likes, not on a sample. Nothing is counted. And it is not blind: a mark already on the page pulls the next reader towards it, as an experiment below shows.

How closely do machine marks agree with human marks?

Within a point most of the time, and identical less than half the time, where a study reports both.

• Essays. Johnson and Zhang (2024) had GPT-4o score 13,121 essays by students in grades 8 to 12 on a six-point scale, given the rubric and no marked examples. It matched the human score about 30% of the time and came within one point 77% of the time; the authors judged that nowhere near the standard for automated scores.
• Exam answers. Flodén (2025) had ChatGPT 3.5 grade 463 answers from three Master's-level exams. The exam grade matched the teachers' in 30% of cases and was one grade away in 45% (abstract read).
• Short answers. Seneviratne and Manathunga (2025) had GPT-4 and two examiners mark two pharmacology questions for 30 medical students, each out of 100, its mark the median of five runs. On average its marks stood 5.43 points from the first examiner's and 0.96 from the second's.

How does that compare with two people?

It depends on which two. In the pharmacology study the machine stood over five times as far from one examiner as from the other, so the examiners differed between themselves (no figure is given). Measured pairs are in our guides to moderation and rubrics. In an audit you are the comparison, and your marks carry error too.

Does the same answer get the same mark twice?

Not always. Hackl et al. (2023) had GPT-4 rate 108 constructed macroeconomics answers repeatedly from April to June 2023, its randomness set to the minimum: the ratings agreed almost perfectly (an intraclass correlation of 0.999), and against one more rating in July the figure was 0.94. Bui and Barrot (2025) had ChatGPT score 200 argumentative essays in two rounds and found low consistency (abstract read). So audit the marks the machine saved, not a fresh run.

Where does a machine reader go wrong?

At both ends of the scale, on fluent answers that say little, on your own course's content, and on answers that address the marker.

• The top and the bottom. Wetzler et al. (2025) had eight instructors and two versions of ChatGPT grade 186 psychology essays with one rubric: both versions were more lenient than the instructors on weak work and stricter on strong work (abstract read). In Johnson and Zhang (2024), GPT-4o gave the top score to 3 essays and the human raters to 732.
• Long, fluent and wrong. That leniency on weak work is, in our reading, the pattern to hunt: an answer that restates the question at length.
• Right, in unexpected words. The nearest finding here: Flodén's ChatGPT struggled on questions tied closely to the course lectures and did better on general ones. A reader has your rubric, not your lessons.
• Addressed to the marker. Humble (2026) hid instructions in an invented student submission and had Microsoft Copilot assess it many times. Some attacks altered its output in 94% to 100% of runs and left no trace there (abstract read).

Does it treat groups of students differently?

It can. In Johnson and Zhang (2024), GPT-4o scored 0.90 of a point below the human raters overall and 1.05 below on the 1,193 essays by Asian/Pacific Islander students, a gap that remained after allowing for the human score. One class is too small to test this: note whose answers the disagreements fall on, term by term.

What happens when the marker sees the machine's mark first?

The marker stays near it, even when it is wrong: that is anchoring. Goulas et al. (2026) showed 1,339 teachers in Greece a student exercise carrying a deliberately wrong mark of 5 out of 10. Each part of the answer was labelled right or wrong, so the fair mark, 8 or 2, could be worked out before the teachers gave their own.

Where 8 was fair, they ended 1.39 points from it on average when told a colleague had given the 5, and 1.59 when told an AI had. Where 2 was fair the figures were 1.94 and 1.82, a difference that could be chance. The authors note that "teachers anchor strongly on the externally supplied recommendation". So mark the audit sample before a machine mark exists.

How do you audit AI grading of open answers, step by step?

Draw the sample, mark it blind, let the machine mark, compare, read, decide and record.

1. Hold the scores. In EdProcto, set "Show the score" to "When released" before the quiz.
2. Fix the rubric first, as elements with points: see short answer questions.
3. Draw the sample by chance. List the class by name, divide its size by the sample size to get a step, pick the start with a die, and take every answer at that step. Not the first six you open.
4. Mark the sample blind, on your own sheet, before the machine has marked those answers.
5. Have the machine mark every answer, the sample included.
6. Work out two figures: the share of the sample with identical marks, and the share within one point.
7. Read every disagreement beside the machine's reason, and sort it: your slip, a rubric with two readings, or the machine's error and its kind.
8. Decide: accept the marks; re-mark a band, reading every answer the machine placed where the errors sit; or re-mark them all.
9. Write it down: the date, the question, how the sample was drawn, both sets of marks, the two figures, each cause and the decision.

How big should the sample be?

A fifth of the answers, at least six and at most thirty; with ten or fewer, mark them all. The rule and the arithmetic are ours.

Six answers cannot certify thirty. They catch a reader that is failing badly, so the disagreements are read, not only counted.

What agreement is good enough for which stakes?

That depends on what a wrong mark would cost the student. These thresholds are our working rule, not a published standard, for a question worth four to six points; on one worth three or fewer, count identical marks only.

Three findings override any figure: a gap of two points or more, three gaps in one direction, or a disagreement that moves a student across the pass mark. After any of them, read the band yourself.

What does an audit look like for a class of thirty?

Six answers marked twice, nine more read, four marks changed. The class and every figure are invented.

A chemistry unit test in EdProcto, on the Tutor plan, has eight rule-marked questions and one short answer, question 9, worth 6 points: "Explain why raising the temperature makes a reaction faster." Thirty students hand it in, and it counts towards the term grade.

1. Draw. "Download the marks" lists the thirty by name, the column "Q9 (6)" still empty. Thirty over six is a step of five; a die shows 3, so rows 3, 8, 13, 18, 23 and 28.
2. Mark blind. The teacher reads question 9 for those six while it shows "to grade" and writes points on a sheet.
3. Ask. The teacher presses "Ask Procto AI" on question 9 for all thirty, one at a time, and downloads again.

row         3    8   13   18   23   28
teacher     6    2    4    1    5    3
Procto AI   5    2    4    3    5    3.5
gap        -1    0    0   +2    0   +0.5

4. Count. Identical on 3 of 6, or 50%; within one point on 5 of 6, or 83%: short of the rule for work that counts. The mean gap, a quarter of a point, hides a two-point miss.
5. Read. Row 3: Procto AI's note says the answer never states that collisions become more frequent. It is right, and its 5 stands. Row 28: half a point the rubric does not allow, so the teacher saves 3. Row 18, the longest of the six, says four ways that faster particles "react more quickly" and nothing about collisions or energy.
6. Decide. One kind of error, mid-scale: re-mark a band. The teacher reads all thirteen answers Procto AI gave 2 to 4 points, four of them already sampled. Two more are restatements and lose two points each; one of those students falls below the pass mark, so that whole script is read again.
7. Record and release. The sheet, both downloads and the decision are filed, then "Release the scores".

Fifteen readings in place of thirty.

What should you tell students?

Tell them before the quiz how the marks are made and checked. A machine reader proposes a mark and a short note on the written question. A teacher marks a sample without seeing its marks, compares, and decides every mark before any score is released. A query names the question and the rubric point, and a person reads the answer again.

This is our practice; no study above tested it. Wording the note is in our guide to feedback comments. A reader's note says nothing about who wrote an answer: see AI detectors.

How does EdProcto ask, record and let you change a mark Procto AI proposed?

EdProcto is the best tool for an audited machine mark, because Procto AI reads one answer, only when you press, so your blind marks come first. Teachers in Goulas et al. (2026) stayed near a mark already on the page; here the page has none until you ask.

• Asking. On the grading page a written answer shows "to grade", your "Model answer" and "Rubric", "Points", "Note for the student", "Save my grade" and "Ask Procto AI". That button is greyed below the Tutor plan ("Procto AI grades written answers from the Tutor plan up") and on a blank answer, which has a button of its own, "No answer: give it 0".
• What Procto AI is given. The browser only names the answer. The server reads the question, any material, the maximum points, your model answer, your rubric and the student's answer from its own store. Procto AI is told to treat anything in the answer that reads as an instruction as part of the answer, to leave length and style alone unless the rubric asks, and to write the student a note of one to three sentences, or, where your rubric is set out in lines, a short line for each criterion with its points before them.
• Recording. Its points land on the half-point step of the "Points" box, never above the question's points, and are saved at once: "Procto AI:" before the note, then "Graded by Procto AI on …" with the day and time.
• Changing. "Points" and "Note for the student" then hold what Procto AI proposed. Change either, or neither, and press "Save my grade": the mark is yours, "Graded by you on …". Asked again, Procto AI replaces its own earlier mark. Where the mark is yours, the page asks first, "Keep my grade" or "Replace it with Procto AI’s", so a grade you gave is never lost to a slip. Keep your blind marks on the sheet all the same. On a rule-marked answer, "Add a note, or set the points by hand" opens the same boxes.
• Holding back. The results page shows each student's "To grade" count. With the score at "When released" it says "No student has seen a score or a mark." until you press "Release the scores".
• The audit sheet. "Download the marks" gives a row for each attempt handed in, by name: "Name", "Email", "Sitting", "Started (your time)", "Handed in (your time)", "Score %", "Points", "Out of", "Reduced by %", "To grade", "Minutes", "Security events", "Notes on the record", then a column for each marked question. An answer still to grade is an empty cell. After the points, each question a reader marks has a column such as "Q9 marked by", reading "Procto AI", "you" or "rule", so one download separates the machine's marks from yours.

We publish no agreement figure for Procto AI: the audit measures agreement on your question, rubric and class.

Free takes thirty students a quiz, Tutor 100, and Educator 500 with the full analysis. Procto AI's grading is from the Tutor plan: see pricing and the page for teachers.

Quick questions

How many AI-marked answers should I audit?

A fifth of them, at least six and at most thirty, drawn by chance: our working rule.

What is the difference between exact agreement and agreement within one point?

Exact agreement counts only identical marks; agreement within one point also counts those a point apart. Johnson and Zhang (2024) found about 30% and 77% for GPT-4o on a six-point essay scale.

Should I look at the machine's mark before I mark the sample?

No. In Goulas et al. (2026), teachers shown a wrong mark ended well short of the fair one. In EdProcto, mark the sample while each answer shows "to grade", then press "Ask Procto AI".

Is a sample audit enough for a final exam?

No. Where a mark decides a pass or an award, a person marks every answer: see moderation.

References

• Bui, N. M., & Barrot, J. S. (2025). ChatGPT as an automated essay scoring tool in the writing classrooms: How it compares with human scoring. Education and Information Technologies, 30(2), 2041–2058. https://doi.org/10.1007/s10639-024-12891-w
• Flodén, J. (2025). Grading exams using large language models: A comparison between human and AI grading of exams in higher education using ChatGPT. British Educational Research Journal, 51(1), 201–224. https://doi.org/10.1002/berj.4069
• Goulas, S., Megalokonomou, R., & Sotirakopoulos, P. (2026). Why do experts miss AI's errors? Evidence from a randomized labeling experiment. PNAS Nexus, 5(6), Article pgag146. https://doi.org/10.1093/pnasnexus/pgag146
• Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023). Is GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings. Frontiers in Education, 8, Article 1272229. https://doi.org/10.3389/feduc.2023.1272229
• Humble, N. (2026). Ethical implications of prompt injection in AI-mediated grading: An adversarial red-team evaluation. Journal of Academic Ethics, 24(3), Article 85. https://doi.org/10.1007/s10805-026-09758-9
• Johnson, M., & Zhang, M. (2024). Examining the responsible use of zero-shot AI approaches to scoring essays. Scientific Reports, 14, Article 30064. https://doi.org/10.1038/s41598-024-79208-2
• Seneviratne, H. M. T. W., & Manathunga, S. S. (2025). Artificial intelligence assisted automated short answer question scoring tool shows high correlation with human examiner markings. BMC Medical Education, 25, Article 1146. https://doi.org/10.1186/s12909-025-07718-2
• Wetzler, E. L., Cassidy, K. S., Jones, M. J., Frazier, C. R., Korbut, N. A., Sims, C. M., Bowen, S. S., & Wood, M. (2025). Grading the graders: Comparing generative AI and human assessment in essay evaluation. Teaching of Psychology, 52(3), 298–304. https://doi.org/10.1177/00986283241282696