A quiz score counts right answers and says nothing about how each was reached. The student who knew, the student torn between two options and the student who picked at random all collect the same mark. So do two students who got it wrong, one who admitted doubt and one who was certain.
Confidence-based marking, also published as certainty-based marking, adds one step: after answering, the student says how sure they are, and the mark depends on both. EdProcto carries the shape as its multiple choice with confidence template.
What is confidence-based marking?
Confidence-based marking scores a question on two things at once: whether the answer is right, and how sure the student said they were. Sure and right earns the most, and sure and wrong loses the most.
Gardner-Medwin (1995), describing an early computer system of this kind used in London medical schools, put the case bluntly: confidently believing a wrong answer is worse than knowing you do not know. A plain score cannot tell those two apart. Nor can it tell knowledge from luck: Burton (2001) worked out how far random guessing lowers the reliability of multiple-choice and true or false tests, and judged many multiple-choice tests too unreliable for their purpose as usually run.
The answer and the rating together give four outcomes.
| The answer is right | The answer is wrong | |
|---|---|---|
| Sure | Knowledge. Move on. | A confident error, usually a misconception. Reteach it. |
| Unsure | Luck, or knowledge that is not yet firm. Ask it again later. | Honest doubt. A gap the student already knows about. |
A plain score folds each column into one number.
How does the scoring reward honest confidence?
The scoring rewards honest confidence by turning a claim of certainty into a bet: more marks if the answer is right, a loss if it is wrong, with the loss sized so that the bet pays only when the student really is that sure.
The scheme that both studies below describe is the one Gardner-Medwin (1995) set out. After each answer the student picks a confidence level of 1, 2 or 3. A right answer earns the level as its mark. A wrong answer earns 0 at level 1, −2 at level 2 and −6 at level 3. Students were told to use level 2 by default, level 3 only above an 80% chance of being right, and level 1 below 67%.
Those thresholds fall out of the marks. The table gives the mark a student can expect on average at each level, calculated for this guide: the chance of being right times the mark for right, plus the chance of being wrong times the mark for wrong.
| Chance the answer is right | Level 1 (1 or 0) | Level 2 (2 or −2) | Level 3 (3 or −6) | Best choice |
|---|---|---|---|---|
| 50% | 0.50 | 0.00 | −1.50 | Level 1 |
| 70% | 0.70 | 0.80 | 0.30 | Level 2 |
| 80% | 0.80 | 1.20 | 1.20 | Level 2 or 3 |
| 90% | 0.90 | 1.60 | 2.10 | Level 3 |
Below a two in three chance, level 1 gives the best return, then level 2 up to 80%, then level 3. A student gains nothing on average by claiming more confidence than they have, or less. Gardner-Medwin calls that a proper marking scheme.
Why the loss has to be bigger than the gain
Each step up the scale is a bet. Going from level 2 to level 3 wins one extra mark if right and loses four more if wrong, which breaks even at odds of four to one: an 80% chance. Going from level 1 to level 2 wins 1 and loses 2, breaking even at two to one, or 67%. The extra loss divided by the extra gain is the odds the student needs.
So the size of the loss gives the word sure its meaning. Make the loss equal to the gain and the bet pays at anything better than an even chance: sure then means only more likely than not. Take the loss away and the top level always has the best expected mark, every sensible student claims it every time, and the rating tells the teacher nothing.
What the scheme does to a guess
The 50% row is a coin toss between two options. Admitted at level 1 it expects half a mark, and claimed at level 3 it expects to lose one and a half. A guess is still allowed, but it has to be owned up to.
What did studies of students using it find?
The published studies found that students judged their own confidence well on average, and that a good many did not want the extra work.
Gardner-Medwin (1995) reported 107 voluntary self-test sessions by medical students on a bank of 376 true or false questions in physiology, in the seven weeks before their exams. Of 6,950 questions shown, 7% were skipped. The answers were right 83% of the time at level 3, 71% at level 2 and 56% at level 1, each inside the band where that level is the best choice. That match is called calibration: on average these students knew how much they knew.
Schoendorfer and Emmett (2012) tried certainty-based marking on optional weekly practice quizzes for 400 second-year medical students at the University of Queensland. Take-up was low: 15%, 11%, 9% and 8% of the cohort used the quizzes over the four weeks of the trial, three of them certainty-marked, against 46%, 44%, 44% and 40% for the ordinary quizzes in the module before. Of the 39 students who answered the survey, 57% agreed the method was useful, 67% said it took more time, and 86% thought it suited revision better than an examination.
Both studies are small, come from medical schools and relied on volunteers, and the first reports data its author calls preliminary. Neither says what a class of thirteen-year-olds would do.
Why is the confident wrong answer the one to find?
The confident wrong answer is the one to find because it is the only kind of error a student will not go back to: nothing has told them there is anything to fix.
Curtis et al. (2013) recorded the confidence of 104 third-year dental students on each of 20 multiple-choice questions about implant dentistry. They split the wrong answers into uninformed (wrong and not confident) and misinformed (wrong but confident). Misinformed answers were the more common, 22% against 8%, and turned up more on complex questions than on factual ones. Staff also rated each wrong answer for the harm it could do in clinical care, and confidence did not fall as that harm rose.
Butterfield and Metcalfe (2001) expected errors made with high confidence to be the hardest to correct and found the reverse: once people were given the right answer, the errors they had been most sure of were the most likely to be put right at a later retest. They called such errors hypercorrected. That was an experiment, not a term of teaching, so read it modestly: a confident error followed by the correct answer is a good moment to teach, and one followed by nothing stays where it was.
What are the costs of marking by confidence?
Marking by confidence costs time, an explanation before the quiz, and some purity in the score, which now measures a little of a student's caution as well as their knowledge.
Time. In the Queensland survey, estimates of the extra time ran from 30 seconds to 30 minutes, and the most common answer was about half as long again (Schoendorfer and Emmett, 2012). Set fewer questions for the same sitting.
Explanation. A rating is honest only if students know the rule before they answer. Gardner-Medwin (1995) noted that the principles seemed complicated to some students when first explained and gave no trouble with experience, and in Queensland 52% of those surveyed agreed that it was easy to understand. Give the marks for each rating and one worked example before the quiz, and say plainly that a low rating is safe.
Caution in the score. Two students with the same knowledge, one bold and one careful, can finish with different marks. Schoendorfer and Emmett (2012) record the concern that overconfident and underconfident students may both be disadvantaged, and the reply that such students are meant to become more self-aware through the process. Gardner-Medwin's (1995) figures show how hard this is to untangle: the men in his sample chose level 3 for 58% of their answers and the women for 31%, yet at level 3 the two groups were right about equally often, 82% and 85%. He put most of the gap down to a difference in knowledge in that self-selected sample. One sample cannot settle it.
Unfamiliarity. Queensland's low take-up was on quizzes that counted for nothing, and the authors suggest introducing the method at the start of a programme.
How do you read the four quadrants after the quiz?
Read the four quadrants in the order of what only this shape can show: confident errors first, then unsure right answers, then ordinary gaps.
- Count the confident errors on each question. Several students sure of the same wrong option is a misconception the class shares. Gardner-Medwin's system totalled right and wrong answers at each confidence level for every question, and he noted that the confidently wrong ones can be particularly revealing for teachers.
- Check the question before the class. A wrong key or an ambiguous stem produces the same pattern. EdProcto's results show, in the full analysis, how many students took each option and how those students scored overall, so a "wrong" option favoured by the strongest students points at the question.
- Reteach each confident error with its reason. Show the right answer and say why the wrong one tempts.
- Bring back the unsure right answers. The score says the student knows it and the student says otherwise, so ask the idea again in a later quiz.
- Treat unsure wrong answers as normal teaching. The student already knows the gap is there.
- Tell each student how well they judged. Being right no more often when sure than when unsure means the rating is not yet a judgement.
How does EdProcto mark multiple choice with confidence?
EdProcto is the best tool for confidence-rated questions when they have to come from your own material, because the rating is built into the question's shape and the rule that marks it is one switch, applied identically to every student on the server.
The multiple choice with confidence template is a multiple-choice question with a second step: the student also says how sure they are, from 1 to 5. Put a chapter in the quiz maker on the home page, pick the shape, and Procto AI drafts the questions from it for you to edit. The rules in how to write multiple-choice questions that measure still apply: a rating cannot rescue a question whose wrong options nobody believes.
The rating changes a mark only when the Confidence bonus setting is on. Then a rating of 4 or 5 counts as sure. A right answer earns a tenth of the question's points more at 4 and a fifth more at 5. A wrong answer loses a fifth at 4 and three fifths at 5. A rating of 1, 2 or 3 changes nothing. For a question worth one point:
| Rating | Right answer | Wrong answer | Pays when the chance of being right is above |
|---|---|---|---|
| 1, 2 or 3 | 1.00 | 0.00 | any |
| 4 | 1.10 | −0.20 | two in three |
| 5 | 1.20 | −0.60 | four in five |
Negative marking is a separate setting that takes one over the number of wrong options from any wrong choice, and the two add together.
This is EdProcto's own scale, not the 1, 2, 3 and 0, −2, −6 scheme set out earlier, but it rests on the same arithmetic and lands on the same two thresholds. Stepping up to 4 wins a tenth and risks a fifth, which breaks even at two to one: a 67% chance. Stepping from 4 to 5 wins another tenth and risks another two fifths, which breaks even at four to one: 80%. So a student does best by rating honestly: unsure below two chances in three, 4 from there to four in five, 5 beyond. The swing is gentler than in the published scheme. A right answer gains a fifth of a question at most, so a total stays close to the plain score, and a confident error costs three fifths of the question, enough to be felt and worth explaining to the class before the quiz.
With the bonus on, negative marking off and no other deduction set, every mark names its quadrant: more than the question's points is sure and right, exactly the points is right without being sure, zero is wrong without being sure or left blank, and below zero is sure and wrong.
Free covers the core set of templates, with marking by rule and a result for every question. Tutor opens all 111 templates, this shape among them, and Educator adds the full analysis. The pricing page lists the rest.
When should you leave the confidence rating out?
Leave the confidence rating out when students have not practised with it, when the right answer is open to argument, or when a simpler repair for guessing will do.
| Situation | Better choice |
|---|---|
| The quiz counts and the class has never met the method | A plain question now, and practice rounds first |
| The right answer is open to argument | Fix the question. A confident, able student who reads it differently takes the heaviest loss |
| All you want is an honest "I do not know" | EdProcto's yes, no or maybe shape, which gives a third answer and has no scale to explain |
| The only worry is the fifty-fifty guess | More statements or negative marking: see when true or false questions work |
Quick questions
Is certainty-based marking the same as confidence-based marking?
Yes. The two names describe one method: Gardner-Medwin (1995) wrote of confidence assessment and Schoendorfer and Emmett (2012) of certainty-based marking, and both describe the same three levels and marks.
Is confidence-based marking the same as negative marking?
No. Negative marking takes marks off for any wrong answer. Confidence-based marking takes more off for a wrong answer the student claimed to be sure of, and lets an unsure student limit the loss. EdProcto has a separate setting for each.
Can a confidence rating be used on true or false questions?
Yes. The 1995 data came from true or false questions in physiology. In EdProcto, write the statement as a multiple choice with confidence question whose only two options are True and False.
Should a confidence-marked quiz count towards a grade?
Not at first. Most of the Queensland students surveyed wanted it for revision, not examinations, and Gardner-Medwin's system was used only for self-assessment and tutorial exercises when he described it. Let a class practise first on quizzes that do not count.
References
- Burton, R. F. (2001). Quantifying the effects of chance in multiple choice and true/false tests: Question selection and guessing of answers. Assessment & Evaluation in Higher Education, 26(1), 41–50. https://doi.org/10.1080/02602930020022273
- Butterfield, B., & Metcalfe, J. (2001). Errors committed with high confidence are hypercorrected. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(6), 1491–1494. https://doi.org/10.1037/0278-7393.27.6.1491
- Curtis, D. A., Lind, S. L., Boscardin, C. K., & Dellinges, M. (2013). Does student confidence on multiple-choice question assessments provide useful information? Medical Education, 47(6), 578–584. https://doi.org/10.1111/medu.12147
- Gardner-Medwin, A. R. (1995). Confidence assessment in the teaching of basic science. Research in Learning Technology, 3(1), 80–85. https://doi.org/10.3402/rlt.v3i1.9597
- Schoendorfer, N., & Emmett, D. (2012). Use of certainty-based marking in a second-year medical student cohort: A pilot study. Advances in Medical Education and Practice, 3, 139–143. https://doi.org/10.2147/AMEP.S35972