Can AI write good quiz questions? What to check before using them

The EdProcto editorial teamWritten and checked against the research it cites. Last checked October 10, 2026. How the guides are made

Can AI write good quiz questions? Often, and that is the difficulty: a set in which most questions are sound is the set nobody reads closely. A language model drafts. Whether each question is true, at the level you wanted and about what you taught is for a person to decide.

EdProcto, which publishes this guide, drafts questions from your material with Procto AI and shows each one's key, source line and explanation. The guide is for a teacher, lecturer or trainer deciding which drafts to keep.

Every study below was read at its source on 10 October 2026. The models are already old: treat the figures as reasons to check.

What did experts find when they rated machine-written questions?

Experts passed most machine-written questions and found the same few faults in the rest. Our guides to a quiz from a PDF and a quiz from your own handout report other studies.

Wrong and second keys. Klang et al. (2023) had GPT-4 write a 210-question medical examination in March and April 2023 from examples of an existing one, and five specialists reviewed it without being told its source. One question had a completely mistaken answer, 13 had more than one possible answer, and about 15% needed some correction. Such a paper, they conclude, "necessitates rigorous inspection by specialist physicians".
Level, and answers not in the text. Elkins et al. (2023) had InstructGPT write 612 open questions from 68 Wikipedia passages on biology and machine learning, each aimed at a named level of Bloom's taxonomy or of difficulty; 19 people with teaching experience rated them. About 97% were relevant, and the average question was judged usable with minor edits or none. But 7% to 9% had no answer in the passage, and only 67% to 69% sat at the level requested: 83% to 92% of those meant to test remembering, 36% to 40% of those meant to have students create something.
Level, with no text supplied. Scaria et al. (2024) gave five models only the topics of a graduate data science course: 2,550 questions, each aimed at one Bloom level, rated by two experts who teach the subject. Both called 78% high quality, and 65.56% of those matched the level asked for.
The source line. Biancini et al. (2025), in a paper first published in 2024, had Llama 2, Mistral and GPT-3.5 write questions from a Wikipedia article on the causes of the Second World War, each with the passage its answer came from. Twenty-one school and university educators rated them. The models differed significantly on keeping to the text, and two sometimes named a sentence's position in place of the sentence.

The raters were few, from two to 21, and no study here watched a school class.

How do machine-written questions behave with real students?

On average, like questions written by people. Three studies gave students machine-written multiple-choice questions and worked out each one's difficulty (the share who got it right) and discrimination (whether stronger students got it right more often).

Isley et al. (2025), a preprint, built ten-question exams from each instructor's course materials with o3-mini, which also judged and discarded drafts; the ten hardest of every twenty were kept. In spring 2025, 1,208 students in 71 college classes sat them, and 478 in 20 statistics classes sat questions from a published practice examination. Students got about 60% of the machine questions right against 39%, and average discrimination, on an item-response scale, was 1.3 against 1.2.
Kıyak et al. (2024; abstract read) had ChatGPT write ten case-based questions, and an expert panel put two, unedited, into an examination for 99 medical students. Both discriminated acceptably (0.41 and 0.39). In one, three options were each chosen by fewer than 5% of the students.
Hölzing et al. (2026; abstract read) set 15 questions from a custom GPT-4-based model, adapted with lecture slides and past questions, beside faculty questions in an examination for 157 students. Difficulty (0.79 against 0.81) and point-biserial discrimination (0.17 against 0.19) did not differ significantly, and the authors judge both sets modest.

So the finding is parity with human writers, whose classroom questions are often flawed, as how to write multiple-choice questions that measure reports.

Does it matter whether questions come from your text or the model's own knowledge?

It decides what you can check. Which gives fewer faults is unsettled: two searches on the day of writing found no experiment that ran one model both ways and had experts count the errors.

From a supplied text, each answer can be checked against it. In Elkins et al. (2023), 91% to 93% of questions had an answer in the passage.
From the model's own knowledge, as in Scaria et al. (2024), there is no page to point to: the check is your own command of the subject.
A source line helps only when you open the source. In Biancini et al. (2025) the line itself could be wrong.

What about open questions and rubrics?

Less is known. The 612 questions in Elkins et al. (2023) were open ones, and their weak point was level, not relevance.

On rubrics we found one study. In Aksu Dunya et al. (2026; abstract read), 30 trainee English teachers each wrote one analytic writing rubric alone and one with ChatGPT's help. Ten expert raters scored the assisted rubrics significantly higher, and the trainees reported generic content and a poor fit with their teaching aims.

We found no study that tested a machine-written rubric against the marks it produced, so read a drafted one as you would a new colleague's. Grading open answers covers the marking.

How do you check a drafted question against its source?

Read it with the source open and ask six things in order. Repair or delete at the first no.

Ask	How to test it	The fault it catches
1. Is the key right, and the only right answer?	Answer before you look at the key, then defend every other option	A second right answer: 13 of 210 in Klang et al. (2023)
2. Is every wrong option wrong, and plausible?	Name the mistake that leads to each	Options nobody chooses
3. Is the answer in the material?	Put a finger on the sentence	No answer in the text: 7% to 9% in Elkins et al. (2023)
4. Does it test what you taught?	Match it to an aim of the lesson	A true question about a passing remark
5. Is the level right?	If finding one sentence answers it, it is recall	About a third missed the level asked for
6. Is the wording clean?	Read it aloud: one question, no negative, no clue in grammar or length	A stem that gives the key away

Our guides to distractors and questions above recall go further on points 2 and 5.

How many should you draft for each one you keep?

Draft about twice as many as you need when the level matters, and half as many again for recall practice. That is our working rule, not a research finding. It leans on three figures above: about 15% needing correction in the kindest study, about half of the questions in Scaria et al. (2024) being both sound and at the level asked for, and Isley et al. (2025) keeping ten of every twenty.

What do you do after the first sitting?

Read the results question by question before the set is used again, because a class finds what a reader misses. EdProcto's results give the share right for each question on every plan. Four patterns name a failed question:

Most of the class wrong, your best students among them. Suspect the key first: see the question with the wrong key.
Stronger students wrong more often than weaker ones. A second right answer, or an unclear stem.
An option nobody chose. It was never plausible, as in Kıyak et al. (2024).
Nearly everyone right. Often recall, or a key given away.

Correct or replace the question, then mark again. Item analysis says how far one class's figures can be trusted.

What stays your responsibility

Four things, whoever drafted the question: that the keyed answer is true, that the question is about what you taught, that the material was yours to use (your own notes, or a published paper only where its owner's terms allow), and every mark that follows.

How careful should you be at each level of stakes?

Match the checking to what the score will be used for.

Stakes	Before students see the questions	After the sitting
Practice with no mark	The six-point read	Fix what the results show
A graded class quiz	The six-point read; sit it yourself; a colleague answers without the key	In EdProcto, set "Show the score" to "When released" and read the results first
A final exam, a certificate, a hiring decision	A second subject expert, and a trial on a comparable group, as for a question bank	Item analysis; rewrite or retire what fails

A worked example: three drafted questions through the checklist

These three drafts are illustrations: we wrote them, faults included, and they are not the output of any tool. The source is a paragraph of a school science handout.

Series and parallel circuits. In a series circuit the lamps are connected one after another in a single loop, so the same current passes through each. If one lamp fails, the loop is broken and every lamp goes out. In a parallel circuit each lamp is on its own branch. If one lamp fails, the others stay lit.

Draft	Fails at	Repair, key first
1. Which is true of a series circuit? The same current passes through each lamp (keyed); if one lamp fails, every lamp goes out; each lamp is on its own branch	Point 1: the second option is right too	One lamp of three in series fails. What happens to the others? Both go out; both stay lit; both get brighter
2. In a parallel circuit, how does the voltage across each branch compare with the supply's? The same (keyed); shared between the branches; higher	Point 3: true, and the paragraph never mentions voltage	Delete it
3. In which circuit are lamps connected in a single loop? Series (keyed); parallel; solar; magnetic	Points 2 and 5: two options nobody would choose, and plain recall	A string of lights goes dark when one bulb blows. How are the bulbs connected? In series; in parallel; it cannot be told

Draft 2 is the instructive one: its key is sound physics, so only the source shows the class was never told.

How does EdProcto draft questions, and what does it show you to check?

EdProcto is the best tool for this job when the questions must come from your own material, because its request and its screen follow the checklist: Procto AI is asked for one defensible right answer and for the place in your material where it is found, and each draft opens with its key, that source line and its explanation in boxes you can change. In Klang et al. (2023) it took a reader to find 13 second answers in 210 questions.

Asking. In the builder, "Write with Procto" leads to the Procto AI pane. It takes a file ("Drop a PDF, a Word file or a text file") or pasted text ("Read it") and lists "Concepts it keeps returning to" to tick. You set the "Shape", such as multiple choice or short answer, "How many" (1 to 40) and a difficulty from "Recall and recognition" to "Evaluation and synthesis". Under "Answers drawn from", "Your material only" is offered once the material has been read; the other choice is "General knowledge".

The request. For multiple choice: "Each question has three to five options and exactly one right one; the wrong ones are plausible, of the same length and form as the right one, and each wrong one is a mistake a student really makes." With "Your material only", every question is to be answerable from the material alone, with the place given (a page, a heading or a paragraph). With "General knowledge" it is asked to leave the place empty where the answer is not in the material, so an empty source box marks a question to check from your own knowledge. Each question is also to carry an explanation, a topic and a label of easy, medium or hard.

What is kept. The form is tested, not the truth: a draft with no prompt is left out, and so is a multiple-choice draft without exactly one option marked right. The rest arrive under a line such as "10 questions added, every one yours to edit."

What you see. Each draft opens in the editor with "Prompt", "Options · mark the right one", "Difficulty", "Topic (for the results by topic)", "Where the answer is in the source (optional)" and "Explanation shown to the student afterwards". Change any box, or press "Remove".

Afterwards. "Got it" is the share right for each question. On the Educator plan the full analysis adds "Separates", the takers of each option, and sentences such as "Check the key on Q1: the stronger students got it wrong more often than the weaker ones." Each is a reason to reread, not a verdict. After a key is corrected in the builder, "Re-mark this quiz" marks the handed-in attempts again.

Plans. On Free, Procto AI writes 20 questions a month, for thirty students a quiz; on Tutor 50 a month, for 100. Educator has no monthly cap, within a daily ceiling, 500 students a quiz and the full analysis. See pricing.

Quick questions