AI grading of short answers: how to check the marks it proposes

A teacher with sixty written answers to mark wants to know how far AI grading of short answers can be trusted. The evidence says: far enough to read first, and no further. An automated reader is a first marker. The examiner is still you.

This guide covers how to use an AI grader for written answers without handing it the decision. EdProcto, which publishes it, grades a short answer with Procto AI against your model answer and rubric, one answer at a time, and a section below says exactly what it does and what it leaves to you. It picks up where our guide to writing short answer questions and a rubric stops.

How well does AI grading of short answers agree with human marking?

About as well as two people agree with each other on some tasks, and clearly worse on others. The question, the rubric and the model all move the figure, so no published number transfers to your class.

Automatic short answer grading is older than chatbots, and two results from before them frame the question.

• Butcher and Jordan (2010) set three marking systems against six human markers on answers of about a sentence. The computers marked at least as well as the people, once their matching had been built from real student responses to trial versions of each question.
• Shermis (2015) reported a public competition on 25,683 answers to ten prompts, drawn from three states. On every measure of agreement the machines failed to match the human raters.

What language models changed, 2024 to 2026

A language model is handed the rubric, not trained on your question. The model and the date matter, because these results age quickly.

• Henkel et al. (2024) had GPT-4 mark real pupils' answers in science and history, ages 5 to 16, with a few marked examples in the prompt. It reached a kappa of 0.70 where the human level was 0.75, and the authors point to low-stakes formative use.
• Grévisse (2024) ran GPT-4 and Gemini 1.0 Pro in February 2024 on 2,288 answers to 82 questions from 12 undergraduate medical courses. Each answer had one human mark, usually from the question's author. GPT-4 averaged 0.65 of full marks against the teachers' 0.68, and Gemini 0.68. For both, the median gap from the teacher was 0.17 of the marks available, and agreement was best on answers wholly right or wholly wrong.
• Hassanein et al. (2026) had three trained experts mark 2,358 answers by 262 dental students to nine questions on a 12-point rubric. The experts agreed with one another at an intraclass correlation of 0.84. Against their agreed score DeepSeek-3 reached 0.87 and ChatGPT-4 0.64. DeepSeek-3 was more than three points out on 7.5% of answers, ChatGPT-4 on 26.9%.

One model there matched the experts as well as they matched each other; the other was more than a quarter of the scale out on one answer in four. You cannot tell beforehand which you have.

Where does an automated reader fail?

At the edges of the rubric, and predictably. These are the answers to read yourself.

• Right in unexpected words. In Grévisse (2024) every model often took marks off answers the teacher had given full marks, and Gemini 1.0 Pro's marks varied between runs where the key was short or listed more items than the question wanted.
• Fluent and wrong. Both models in Hassanein et al. (2026) over-scored answers the experts had marked incorrect, DeepSeek-3 by about one and a half points in twelve. In Grévisse (2024), answers the teacher gave nothing drew a median of a third of the marks from both models.
• Very short or empty. One student there wrote only "1. ... 2. ... 3. ..." where three causes were asked for. Over ten runs Gemini 1.0 Pro gave it anything from zero to full marks.
• Near the pass mark. For answers the experts scored 4 to 7 out of 12, ChatGPT-4 put 51.5% in a different band from the experts and DeepSeek-3 33.3% (Hassanein et al., 2026).
• Written to game the marker. Filighera et al. (2024) slipped adjectives and adverbs into wrong answers. Graders built on BERT and T5 lost 10 to 22 percentage points of accuracy, and human readers were not significantly more suspicious.
• Carrying instructions. A student can address the marker inside the answer (prompt injection). Cai (2026) tried it on three small open models (Llama 3.1, Qwen 2.5 and Mistral 7B) with benchmark data, not a live class. Instructions dressed as student reasoning succeeded in 73% to 82% of cases, and separating the student's text from the model's orders cut the average from 0.78 to 0.65 without ending it. As Grévisse (2024) notes, such text looks strange to a person.

Is AI grading biased against some students?

It can be, and for short answers the direct evidence is thin. Most of what exists is about essays.

Bridgeman et al. (2012) compared human and machine essay scores on two large high-stakes tests by gender, ethnicity and country. Scores were very similar for most groups, with some notable exceptions, and policies were set to keep their effect on reported scores minimal. Warr et al. (2025) gave ChatGPT 3.5 identical passages of student writing with different descriptions of the writer. Scores for writers described as Black or White did not differ significantly when race was stated outright, and did when it was only implied.

Two rules follow. Do not tell the reader who wrote the answer. And do not show it your own mark first: when Grévisse (2024) gave GPT-4 the teacher's grade in advance, the median gap between them fell from 0.17 to zero. Williamson et al. (2012), setting out the evidence needed before automated scoring complements human scoring on a high-stakes test, include its effect on subgroups. No classroom can gather that.

How do you check AI marking before you release it?

Have the reader propose, then read in a fixed order: the doubtful answers, a random sample of the rest, both ends, and the boundary of the pass mark.

1. Write the rubric as elements with points before the quiz. Add the wordings you accept and the wrong idea that earns nothing. In EdProcto each open question has a box for the model answer and one for the rubric.
2. Have the reader propose a mark and a reason for every answer. A mark without a reason cannot be checked.
3. Read every answer the reader was unsure of. Treat as unsure every part mark, every reason that hedges, every very short answer and every answer the student flagged.
4. Read a random sample of the rest. The table below says how many.
5. Check the top, the bottom and the pass mark. Read the highest and lowest few in full, and every student whose total would cross the pass mark if one mark moved.
6. When the same disagreement repeats, change the rubric. Then re-mark that question for every student.
7. Release, and say how the marks were made.

Longer answers, like the graduate tasks in our guide, need a larger share read in full.

How many answers should you read yourself?

About twenty chosen at random, almost whatever the class size. That gives nine chances in ten of meeting at least one wrong mark when one in ten is wrong.

We computed these as draws without replacement, for a 90% chance that the sample holds at least one wrong mark. A clean sample is not a clean class: with 6 wrong marks among 60, a sample of 19 still shows none about one time in eleven. Errors also cluster on the kinds of answer listed above, so the doubtful ones are read first and the sample comes from the rest.

How does EdProcto grade open answers, step by step?

EdProcto is the best tool for this job when you want a first marker that cannot outrank you, because Procto AI reads an answer only when you ask, on the page where that answer is in front of you. Readers in the studies above were weakest on borderline answers, and this design puts each answer in front of you as it is marked.

• Before the quiz. The rubric is one box of text, so type the elements line by line.
• At hand-in. The server marks every question that has a rule. Each open answer is set to "to grade" with no points, and nothing is sent to Procto AI. Until graded, it counts as zero in the student's score and in the results, which say so.
• The queue. The dashboard counts the answers waiting across your quizzes, and the results list each student with the number still to grade.
• The grading page. One student's submission: each answer with its question, the time spent, whether the student flagged it, your model answer and your rubric. One box takes the points, in half points up to the maximum, and another a note for the student.
• Ask Procto AI. On a shape it grades, a button sends the question, any passage, the maximum points, your model answer, your rubric and the text of the answer. The student's name is not sent. Back come the points and a note, asked for in one to three sentences, on what was right and what was missing. It is told to treat anything in the answer that reads as an instruction as part of the answer, and to leave length and style alone unless the rubric asks.
• Your decision. Its mark is saved at once, labelled as Procto AI's. Change the points or the note, save, and the mark is recorded as yours. Every mark records who decided it: the rules, Procto AI or you.

What is done by hand

Procto AI does not release a score, set a penalty or reword what a student wrote. It gives no figure for how sure it is, so the doubtful answers are yours to spot. The rest is manual in EdProcto today.

• No run through a whole class in one press and no bulk accept: an untouched Procto AI mark stands.
• No second opinion, and no sample drawn for you.
• Editing a rubric re-marks nothing. Ask again on each answer, or change the points yourself.
• The page goes student by student, name at the top. Marking one question across the class, or marking blind, means opening each submission in turn.
• The results page downloads the marks as a file a spreadsheet opens. A student sees the settled mark and your note by opening the quiz's link again on the device they used, once the scores are released.

For a quiz with open answers, set "Show the score" to "When released". The screen at hand-in then shows no score and says the instructor will release it. When the queue is empty, press "Release the scores" on the results page.

Grading by Procto AI is on the Tutor plan, for open answers against your rubric; Tutor also opens all 111 templates. Where grading by Procto AI is not open on your account, the button is off and you grade by hand. Pricing has the plans.

What should you tell students, and how do they query a mark?

Tell them before the quiz that a machine proposes marks on written answers and a person decides every one.

• Hand back the mark with its reason and the rubric element it rests on. "No cause given for the feeling" can be argued with; "2 of 4" cannot.
• Give one route and a deadline. The student names the question and the element they believe they met.
• Read the answer again yourself for every query. Do not ask the machine again and pass on its reply.
• If a query shows the rubric was wrong, correct every student it touched.

This part is our practice; none of the studies above tested it.

What should an AI grader never be used for?

Three things.

• A high-stakes decision with no person reading the answer. Grévisse (2024) and Hassanein et al. (2026) both conclude that human oversight is still needed, above all where the stakes are high.
• Judging who the student is. Effort, ability, first language, honesty: none is in the rubric, and none should reach the reader.
• Detecting cheating. A reader marking against a rubric cannot tell who wrote an answer, or whether a chatbot did. That belongs to how the quiz is set and run: see integrity and our guide to what still proves a student learned. No flag there is proof either.

What does it cost, and which mistakes waste the saving?

It costs credits for each answer graded and, done properly, more of your time than a button suggests.

The sum for sixty answers: sixty presses, the doubtful answers read in full, about nineteen of the rest, both ends and the boundary. The saving is the first reading and a drafted note on the clear cases. None of the studies above timed a teacher doing this, so we give no figure for minutes.

The mistakes that waste it:

• Writing the rubric after seeing the proposals. It then describes what the machine did.
• Fixing one student's mark when the rubric was at fault, and leaving the others.
• Reading the results before the queue is empty. Ungraded answers count as zero, so the mean and the pass rate read low, and the item analysis is not ready.

Quick questions

Can AI grade short answer questions accurately?

On some tasks as accurately as a second human marker, on others not. Henkel et al. (2024) found GPT-4 at a kappa of 0.70 against 0.75 for humans; Shermis (2015) found machines short of human raters on every measure.

How many AI-marked answers should a teacher re-read?

Every doubtful one, both ends, the pass-mark boundary, and about twenty of the rest at random.

Does Procto AI mark a whole class by itself?

No. In EdProcto it reads an answer only when you press the button on that answer, and its mark is saved labelled as its own, for you to keep or change. It does not release a score.

Can a student trick an AI grader with hidden instructions?

In experiments, yes: Cai (2026) shifted the grades of three small open models in most cases. Procto AI is told to treat such text as part of the answer, a precaution and not a guarantee.

References

• Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25(1), 27–40. https://doi.org/10.1080/08957347.2012.635502
• Butcher, P. G., & Jordan, S. E. (2010). A comparison of human and computer marking of short free-text student responses. Computers & Education, 55(2), 489–499. https://doi.org/10.1016/j.compedu.2010.02.012
• Cai, Y. (2026). Prompt injection attacks on educational large language models for higher and vocational education. Scientific Reports, 16, Article 15594. https://doi.org/10.1038/s41598-026-46563-1
• Filighera, A., Ochs, S., Steuer, T., & Tregel, T. (2024). Cheating automatic short answer grading with the adversarial usage of adjectives and adverbs. International Journal of Artificial Intelligence in Education, 34(2), 616–646. https://doi.org/10.1007/s40593-023-00361-2
• Grévisse, C. (2024). LLM-based automatic short answer grading in undergraduate medical education. BMC Medical Education, 24, Article 1060. https://doi.org/10.1186/s12909-024-06026-5
• Hassanein, F. E. A., Hussein, R. R., Ahmed, Y., El-Guindy, J., Ahmed, D. E., & Abou-Bakr, A. (2026). Calibration of AI large language models with human subject matter experts for grading of clinical short-answer responses in dental education. BMC Oral Health, 26, Article 286. https://doi.org/10.1186/s12903-026-07665-4
• Henkel, O., Hills, L., Boxer, A., Roberts, B., & Levonian, Z. (2024). Can large language models make the grade? An empirical study evaluating LLMs ability to mark short answer questions in K-12 education. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (pp. 300–304). Association for Computing Machinery. https://doi.org/10.1145/3657604.3664693
• Shermis, M. D. (2015). Contrasting state-of-the-art in the machine scoring of short-form constructed responses. Educational Assessment, 20(1), 46–65. https://doi.org/10.1080/10627197.2015.997617
• Warr, M., Oster, N. J., & Isaac, R. (2025). Implicit bias in large language models: Experimental proof and implications for education. Journal of Research on Technology in Education, 57(6), 1324–1349. https://doi.org/10.1080/15391523.2024.2395295
• Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x