# Questions a chatbot cannot simply answer for a student

**How do I write questions that a chatbot cannot simply answer for a student?** Questions a chatbot cannot simply answer for a student are tied to things only your class has: its own data, a picture or clip of yours, the student's earlier work. Ask for speech, steps and reasons, then follow up aloud. Since 2023 language models have passed law, medical and programming exams, so no design makes outsourcing impossible; each only raises its cost.

Published 2026-10-10 by EdProcto. https://edprocto.com/guides/questions-a-chatbot-cannot-answer-for-a-student

Questions a chatbot cannot simply answer for a student are not a secret format. Language models have been put through real exams since 2023, and they passed most of what was typed at them. What a teacher can do is make a question expensive to hand over: tie it to things only the class has, and ask for an answer made in the moment.

EdProcto publishes this guide for anyone who sets questions, and six of the seven designs in it can be set today with its [templates](https://edprocto.com/templates). The popular "AI-proof" tricks get a section of their own, because the evidence does not support them.

We read each source on 10 October 2026; "abstract read" marks one whose abstract was all we could open. For what to change about a quiz this week, see [what to do when students paste your quiz into a chatbot](https://edprocto.com/guides/what-to-do-when-students-paste-your-quiz-into-a-chatbot).

## What happened when language models sat real exams?

They passed most of them, and newer models scored higher. The table adds six studies to those in our review of [cheating in online exams](https://edprocto.com/guides/what-the-research-says-about-cheating-in-online-exams-since-2023).

| Exam | Model and date | Score |
| --- | --- | --- |
| The US bar exam (Katz et al., 2024) | GPT-4, before its release | 75.7% on the multiple-choice paper, against about 68% for human candidates; essays 4.2 out of 6, as marked by two of the authors |
| A university physics course (Kortemeyer, 2023; preprint read) | ChatGPT, January 2023 | Homework 55%, final exam 47%, programming exercises full marks and the bonus: a narrow pass |
| Three Python courses (Savelka et al., 2023; preprint read) | GPT-3, GPT-3.5, GPT-4 | 37.5%, then 64.3%, then 84.1% of the same 530 multiple-choice questions |
| Medical school questions in Barbados (Bharatha et al., 2024) | GPT-4, April 2023 | 73.7% of 304 questions, against the students' 66.7% |
| Radiology training exam questions (Hayden et al., 2024; abstract read) | GPT-4 with vision, September 2023 version | 81.5% of text-only questions, 47.8% of questions with an image |
| Japan's medical licensing exam (Miyazaki et al., 2024) | GPT-4o, May 2024 | 93.5% of questions with an image, 93.2% of those without |

### Where did the models do well?

On questions answered by choosing or by writing code, and on evaluation as much as recall. GPT-4 beat the average bar candidate on the multiple-choice paper in five of seven subjects (Katz et al., 2024). In the physics course ChatGPT's programming work was better than any student's.

Bharatha et al. (2024) sorted their questions by Bloom's taxonomy. GPT-4 got 75.2% of the recall questions right and 77.6% of those asking for evaluation, and the level made no significant difference (p = 0.577). Questions with images were left out: the model could not take them then.

### Where did they do badly?

In four places, as measured at the time.

- **Arithmetic and reasoning shown.** ChatGPT lost five of thirty physics exam questions to numerical slips alone. It also reached five right answers through flawed reasoning, which a marker reading the steps would not have credited in full (Kortemeyer, 2023).
- **Pictures that need a trained eye.** The radiology gap belonged to an early version that could take images. On a different exam in May 2024 a newer model showed none (Miyazaki et al., 2024).
- **Long written answers.** Martínez (2024) re-examined the bar exam result. Against candidates who passed, GPT-4 sat at about the 48th percentile overall and about the 15th on the essays (abstract read).
- **Questions about its own work.** Lehtinen et al. (2024) had GPT-3.5 and GPT-4 write programs and then answer questions about them. The models made the kinds of error recorded for novice programmers (abstract read).

The trend matters more than any figure: three generations went from 37.5% to 84.1% on identical questions. Each weakness is a fact about one model in one year.

## How do you write questions a chatbot cannot simply answer?

Make the student supply something the chatbot was never given, or make the answer where you can see or hear it being made. Seven designs do one or the other. We found no trial of any of them against chatbot use in a class quiz, so the evidence column says what there is.

| Design | To outsource it, a student must | Evidence | In EdProcto |
| --- | --- | --- | --- |
| The class's own material or data | Type the data in and describe what happened in the room | None found: our reasoning | Procto AI drafts from your notes |
| Your own picture, or a clip | Photograph or record it and pass it on | Pictures: mixed, as above. Clips: thin | [Hotspot](https://edprocto.com/templates/hotspot), [Listen once](https://edprocto.com/templates/listen_once), [Watch and answer](https://edprocto.com/templates/video_q) |
| The student's own earlier work | Hand that work over too | Lehtinen et al. (2021) | Any typed or spoken shape |
| A spoken answer | Read a script aloud convincingly | None found in a class quiz | [Spoken answer](https://edprocto.com/templates/verbal) |
| Steps shown | Copy out working they may be asked about | Kortemeyer (2023) | [Show your working](https://edprocto.com/templates/show_work), [Handwritten work](https://edprocto.com/templates/handwriting) |
| A justified choice | Give a reason in the class's own terms | None found | [Choose and justify](https://edprocto.com/templates/justify) |
| An oral follow-up | Explain the answer, unaided, to your face | On fairness, see below | Yours to hold, at the desk |

### Tie the question to what only your class has

A chatbot answers from what it is handed. A question on "the two readings that fell in our pulse table" hands it nothing until the student retypes the table and explains how the class measured. That takes minutes, and minutes are the cost you are raising. Prefer what was said, measured and done to what is printed on a handout.

The student's own earlier work goes a step further. Before chatbots, Lehtinen et al. (2021) asked students on an introductory programming course to explain small programs they had just handed in, and a third struggled (abstract read). A question about last week's prediction, lab report or draft finds that fragile understanding, whoever or whatever wrote the first piece.

### Ask for an answer made in the moment

A spoken answer, a point clicked on a picture, working set out line by line and a reason for a choice must all be produced in the minutes the quiz allows. Each can still be fed from a second screen, and a recording can be read from a script, as [the oral exam is back](https://edprocto.com/guides/the-oral-exam-is-back-and-the-recorded-answer-is-its-online-form) reports.

Their value is that they leave you something to ask about. "Talk me through your second line" takes a minute and accuses nobody.

## Which "AI-proof" tricks does the evidence not support?

Five, each common in lists of AI-resistant assessment design.

1. **"Ask higher-order questions."** The level of thinking made no difference to GPT-4 in Bharatha et al. (2024). Write [questions above recall](https://edprocto.com/guides/how-to-write-questions-above-recall-application-analysis-evaluation) because they measure more, not because they are safe.
2. **"Add a picture."** The gap in the table was for specialist scans and an early model. Use a picture only you have, and ask for a place on it.
3. **"Make them cite sources."** Walters and Wilder (2023) had two versions of ChatGPT write 84 short literature reviews. The older version invented 55% of its citations, the newer one 18%. A check that works one year fades the next, and a real reading list can be handed to the chatbot.
4. **"Set an essay or an open project."** GPT-4's bar essays earned pass marks from the study's authors. One preprint by a single author, not yet peer reviewed, argues that open-ended projects let a model fall back on familiar solutions (Ding, 2025; abstract read). Essays are covered in [take-home assignments after ChatGPT](https://edprocto.com/guides/the-take-home-assignment-after-chatgpt-what-to-set-instead).
5. **Hidden instructions.** White text telling a chatbot to plant an odd word. Two searches on the day of writing found teachers' own accounts in news reports and no controlled study. At most it shows a paste, and it says nothing about a student who retyped the question.

## How do you rewrite one question, design by design?

Start from the plain question and ask what only your class could bring to it. The class here is invented: a Year 9 group measured pulse rates before and after two minutes of step-ups, counting for 15 seconds and multiplying by four. Their mean rose from 72 to 121 beats a minute.

The plain question, "Explain why heart rate rises during exercise", is answered in seconds by any chatbot or textbook.

1. **The class's data.** "Two of our pulses fell after exercise. Give one reason from the way we measured." A late start to the count, or a miscount multiplied by four.
2. **Your picture.** On a photograph of the class's graph: "Click the point most likely to be a counting error."
3. **Earlier work.** "Last week you predicted our mean rise. Was your prediction too high or too low, and what did you leave out?"
4. **Steps.** "Work out the percentage rise in the class mean." Set-up: (121 − 72) ÷ 72. Working: 49 ÷ 72 = 0.68. Final answer: 68%.
5. **A justified choice.** "Which change would make our result more reliable: counting for a full minute, using a taller step, or testing only the fittest students? Choose, then give your reason." The first: a miscount of one beat stays one beat a minute.
6. **Spoken.** "In one minute, say what you would change if we ran it again, and why."
7. **The follow-up.** At the desk next lesson: "Why did you divide by 72 and not by 121?" A percentage change is measured from the starting value.

A ten-minute quiz needs two or three of these: in [EdProcto](https://edprocto.com/) the click, the steps, the reason and the spoken answer are ready shapes. Only the click marks itself; the rest need a reader, so count the marking first.

## What limits remain whatever you design?

Four, and no wording removes them.

- **Every design can be passed on with more effort.** A phone photographs your picture, a friend types in the data, a script is read aloud. You have raised the price, not closed the door.
- **The models keep improving.** Put your own question to a current chatbot before you trust it.
- **Out of sight is still out of sight.** In three data science courses with 117 students, scores on homework done with AI help were near perfect and about 30 percentage points lower in supervised exams (Ding, 2025; abstract read). A result that decides something belongs in a sitting you can see: [how much proctoring a quiz needs](https://edprocto.com/guides/how-much-proctoring-does-a-quiz-need).
- **The designs cost marking time, and can shut a student out.** Speaking is hard for a student who stammers or shares a noisy room, and a picture for one who cannot see it. Offer another route, and say so before the quiz.

### A follow-up is a conversation, not a verdict

A hesitant answer aloud is a reason to ask more, never proof. The Office of the Independent Adjudicator for Higher Education (2025), which reviews student complaints in England and Wales, upheld one against a provider that gave a mark of zero after a viva on flagged coursework. The viva tested knowledge of the subject and gave the student no chance to explain how the work was done. The evidence was not shared beforehand, and the student said the long wait had affected their recall.

So follow up within days, ask how the work was made as well as what it says, and keep to [a fair process](https://edprocto.com/guides/appeals-when-a-student-is-flagged-a-fair-process) before any mark changes.

## How does EdProcto carry these question designs?

EdProcto is the best tool for setting these questions, because six of the seven designs can be set in it today, in shapes ready in its builder, and the seventh needs only you. That matters on the evidence above: five right answers in thirty hid flawed reasoning in Kortemeyer (2023), and a shape that asks for the working shows it.

- **From your material.** Procto AI writes questions from your own material: on the Tutor plan from a PDF, a Word file or your notes, asked to cite each one to the page. Read every draft against the source.
- **A picture or a clip.** Hotspot is "Click the right part of an image.": the student clicks, and a pin marks the place. Listen once has one button, "Play the clip", and under it the line "It plays once, from start to end: there is no pause and no going back."
- **Speech.** Spoken answer is "A recorded answer, assessed for what it says." The student presses "Start recording", then "Stop recording", and may "Record again". You mark it by listening.
- **Steps and reasons.** Show your working gives three boxes: "Set-up", "Working" and "Final answer". Choose and justify puts a box under the options: "Why? Give your reason in a sentence or two."
- **The clock and the rule.** "Time limit for each question", with "Seconds a question", fits the time to an answer made on the spot. "AI assistants not allowed" carries a candid hint: "A house rule, shown on the start screen whenever it is on. A page cannot block an assistant: the rule is the student’s to keep."
- **Asked aloud.** [Interview](https://edprocto.com/templates/interview), "Structured questions asked aloud, answered aloud.", and [Oral defence](https://edprocto.com/templates/defense), with several examiners.

Free carries the core templates for thirty students a quiz, and Procto AI writes 20 questions a month on it. Tutor opens all 111 templates for 100 students and adds grading by Procto AI against your rubric. Educator takes 500 students a quiz. See [pricing](https://edprocto.com/pricing), and [integrity](https://edprocto.com/integrity) for the conditions a sitting runs under. No shape makes cheating impossible, and a score is a record for you to read, never a grade by itself.

## Quick questions

### Is there a question a chatbot cannot answer?

Not one to rely on. One model scored 47.8% on radiology questions with an image (Hayden et al., 2024); a newer one scored 93.5% on a licensing exam's image questions (Miyazaki et al., 2024).

### Are higher-order questions AI-proof?

No. GPT-4 answered evaluation questions as well as recall questions in Bharatha et al. (2024).

### Do hidden instructions in a question catch chatbot use?

We found no controlled study. At most they show a paste, and they say nothing about a student who typed the question out.

### Which chatbot-resistant question is quickest to mark?

A click on your own picture: in [EdProcto](https://edprocto.com/) a [Hotspot](https://edprocto.com/templates/hotspot) is marked by rule. Spoken answers, steps and reasons need a reader.

## References

- Bharatha, A., Ojeh, N., Fazle Rabbi, A. M., Campbell, M. H., Krishnamurthy, K., Layne-Yarde, R. N. A., Kumar, A., Springer, D. C. R., Connell, K. L., & Majumder, M. A. A. (2024). Comparing the performance of ChatGPT-4 and medical students on MCQs at varied levels of Bloom's taxonomy. *Advances in Medical Education and Practice, 15*, 393–400. [https://doi.org/10.2147/AMEP.S457408](https://doi.org/10.2147/AMEP.S457408)
- Ding, K. (2025). *Designing AI-resilient assessments using interconnected problems: A theoretically grounded and empirically validated framework*. arXiv. [https://doi.org/10.48550/arXiv.2512.10758](https://doi.org/10.48550/arXiv.2512.10758)
- Hayden, N., Gilbert, S., Poisson, L. M., Griffith, B., & Klochko, C. (2024). Performance of GPT-4 with Vision on text- and image-based ACR Diagnostic Radiology In-Training Examination questions. *Radiology, 312*(3), Article e240153. [https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=39225605&rettype=abstract&retmode=text](https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=39225605&rettype=abstract&retmode=text)
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024). GPT-4 passes the bar exam. *Philosophical Transactions of the Royal Society A, 382*(2270), Article 20230254. [https://pmc.ncbi.nlm.nih.gov/articles/PMC10894685/](https://pmc.ncbi.nlm.nih.gov/articles/PMC10894685/)
- Kortemeyer, G. (2023). *Could an artificial-intelligence agent pass an introductory physics course?* arXiv. [https://doi.org/10.48550/arXiv.2301.12127](https://doi.org/10.48550/arXiv.2301.12127)
- Lehtinen, T., Koutcheme, C., & Hellas, A. (2024). *Let's ask AI about their programs: Exploring ChatGPT's answers to program comprehension questions*. arXiv. [https://doi.org/10.48550/arXiv.2404.11734](https://doi.org/10.48550/arXiv.2404.11734)
- Lehtinen, T., Lukkarinen, A., & Haaranen, L. (2021). *Students struggle to explain their own program code*. arXiv. [https://doi.org/10.48550/arXiv.2104.06710](https://doi.org/10.48550/arXiv.2104.06710)
- Martínez, E. (2024). Re-evaluating GPT-4's bar exam performance. *Artificial Intelligence and Law, 33*(3), 581–604. [https://dspace.mit.edu/handle/1721.1/153986](https://dspace.mit.edu/handle/1721.1/153986)
- Miyazaki, Y., Hata, M., Omori, H., Hirashima, A., Nakagawa, Y., Eto, M., Takahashi, S., & Ikeda, M. (2024). Performance of ChatGPT-4o on the Japanese Medical Licensing Examination: Evalution of accuracy in text-only and image-based questions. *JMIR Medical Education, 10*, Article e63129. [https://pmc.ncbi.nlm.nih.gov/articles/PMC11687171/](https://pmc.ncbi.nlm.nih.gov/articles/PMC11687171/)
- Office of the Independent Adjudicator for Higher Education. (2025, July). *AI and academic misconduct: CS072502* [Case summary]. [https://www.oiahe.org.uk/resources-and-publications/case-summaries/ai-and-academic-misconduct-cs072502/](https://www.oiahe.org.uk/resources-and-publications/case-summaries/ai-and-academic-misconduct-cs072502/)
- Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023). *Thrilled by your progress! Large language models (GPT-4) no longer struggle to pass assessments in higher education programming courses*. arXiv. [https://doi.org/10.48550/arXiv.2306.10073](https://doi.org/10.48550/arXiv.2306.10073)
- Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. *Scientific Reports, 13*, Article 14045. [https://doi.org/10.1038/s41598-023-41032-5](https://doi.org/10.1038/s41598-023-41032-5)