How to assess large classes: marking 500 scripts by rule

The usual advice on how to assess large classes is to use multiple choice and accept what is lost. With 500 students, half of that is right: nobody can read 500 scripts in a week, but a paper of recall questions measures the wrong thing.

This guide is for a lecturer in higher education with several hundred students and two or three markers. It builds a fifty-minute paper in which 44 of 50 marks are made by rule.

EdProcto, which publishes this guide, makes those marks on its server at hand-in. Its page for universities shows the rest, and the sections below say what is still done by hand.

What does the research say about assessment in large classes?

It says the marking workload and the slow feedback are real, and that the drift to recall is real but not caused by class size.

Workload and delay. Broadbent et al. (2018) describe a course of 1,500 students in one semester, with several markers and limits of cost and time, where the dilemma is consistent, high-quality feedback. In an introductory biology course, Stanger-Hall (2012) added three or four short written questions to each exam in one section, where 192 students joined the study. One graduate assistant marked them all: multiple-choice results were out within a day, the written ones within a week.

Recall. Momsen et al. (2010) rated 9,713 exam and quiz questions from 77 introductory biology courses and placed 93% at the two lowest levels of Bloom's taxonomy, knowledge and comprehension. But class size did not predict the level (r = −0.097 for 57 courses). Small classes set recall questions too, so the cause is how questions are written, which our guide to questions above recall covers.

Which kinds of thinking can be marked by rule, and which cannot?

A rule can mark any answer that can be compared with a key: a choice, a number, an order, a pairing, a grouping, a word in a gap, a point on an image. It cannot mark an explanation, an argument or the working behind a number.

Part marks on multiple answers, ordering, matching, sorting and best and worst need "Partial credit" switched on. It starts off, and each of those is then all or nothing. Gaps are always credited one by one.

Evidence on the newer shapes is thinner than for multiple choice, and encouraging. Denny et al. (2008) had students put scrambled lines of code in order in first-year programming exams. Scores on those questions correlated notably with scores for writing code, and only weakly with tracing it.

Do multiple-choice and constructed-response questions measure the same thing?

Mostly yes when both formats ask the same question, and less so when the written task is an essay.

Rodriguez (2003) pooled 56 correlations from 29 studies. Where the two formats shared a stem, the mean correlation, corrected for the unreliability of each test, approached one. It was significantly lower where the stems differed, most of all for essays.

Scott et al. (2006) checked the multiple-choice exams of the large introductory physics courses at the University of Illinois. For a subset of students, the ranking by multiple-choice score agreed with instructors' rankings from written explanations (r above 0.94) and from interviews (r = 0.94).

The disagreement is about learning, not measurement. In Stanger-Hall (2012), the section with written questions reported more active study habits, and scored 67.3% on the cumulative final against 63.8% for the multiple-choice section. The sections were not randomised. It is still the best reason we found to keep one written answer: students prepare for the questions they expect.

What must a person still read, and how do you ration it?

A person must read any answer where the reasoning is the point, so ration it to one short written question, marked against a rubric, with a second marker reading a sample.

Bloxham et al. (2016) studied 24 experienced UK assessors in four disciplines, found that they chose, ranked and scored criteria differently, and concluded that some variability is inevitable. Calibration narrows it: Bird and Yucel (2013) trained a large team of laboratory demonstrators with a rubric, sample reports and moderation discussions, after which their marks on first-year biology reports sat significantly closer to an expert's.

How marker reliability is checked

1. Write the rubric as elements with points before the paper is sat. EdProcto's short answer question holds your model answer and rubric, and our guide to short answers and a rubric has a worked one.
2. Calibrate. Every marker marks the same ten answers, and the team settles each difference in the rubric's wording.
3. Mark. Split the 500 between markers at random, not by tutorial group.
4. Second-read a random 50, blind. Count the answers where the two marks differ by more than one.
5. Act on a pattern. If the differences sit with one marker or one rubric element, fix it and re-mark that batch.

These steps are our advice, not a finding. A sample catches drift, not rare slips: 50 scripts from 500 have a 93% chance of including a badly marked one when 25 are bad, and 41% when five are.

Procto AI will grade a written answer against your rubric from the Tutor plan up, within the plan's allowance for the day and the month, but only when you press "Ask Procto AI" on that answer. Here that is 500 presses and 500 marks to check: a queue. Our guide to checking the marks it proposes covers it.

Why is marking by rule harsh on multi-step problems?

Because a rule compares the last number with the key, so a slip in the first line costs every mark, where a human marker charges it once. Markers call that error carried forward: later steps are judged on the student's own earlier figure.

Ashton et al. (2006) describe the result in a Scottish project on school mathematics: a computer awards nothing for a wrong answer that a paper marker would part-credit, so one exam yields different marks in the two media. They conclude that optional steps are needed to mimic partial credit.

Take a three-step calculation worth 6 marks. A student slips in step one, then works steps two and three correctly from the wrong figure.

• On paper, where the slip costs one accuracy mark: 5.
• One final-answer box, by rule: 0.
• Three boxes, each compared with the key: 0.
• Three boxes, each later step starting from a figure you print: 4.

The last line is the remedy. EdProcto's scaffolded sequence fixes the order of the steps and allows no going back, so with one question a page, step two can print the right result of step one. Its rules never compare a step with the student's own earlier figure.

A tolerance softens rounding, not method. EdProcto takes the gap between the student's number and the key as a percentage of the key. With a key of 12.4, full marks within 2% and half within 10%, an answer of 12.6 (1.6% off) earns everything, 13.5 (8.9% off) half and 14.0 (12.9% off) nothing. A new question starts at 5% and 15%, loose for an exam.

Should every student get the same questions?

Yes when the whole class sits at once, with the order shuffled. When students sit at different times, draw each question from a small pool.

Chen et al. (2018) studied a course of 425 students whose computer-based exams ran over several days, and inferred from the pattern of scores that about 5.5% were receiving information from earlier sitters. Those students gained 13 percentage points when everyone got the same problem, even with its numbers changed, and 2 to 3 points with pools of two or four.

Question pools cost fairness: every version must be equally hard. We could not read a study measuring how equal pooled versions turn out, so compare each version's mean afterwards.

In EdProcto, "Shuffle the questions for each student" and "Shuffle the options" start switched on. On the Educator plan, "A different set of questions for each student" has Procto AI write each student their own questions on the same concepts; versions you write yourself are separate quizzes, each with its own code and statistics. You open a quiz with "Publish & share" and end it with "Close the quiz", or set "Open only between two times". No setting makes copying impossible.

How do you assess large classes in fifty minutes? A worked paper for 500

Give 44 of the 50 marks, 88%, to shapes a rule can mark and 6 to one written answer. Every rule-marked question sets a case, a calculation or a judgement; none asks for a definition.

The minutes are our allowances, with two spare. Switch "Partial credit" on. A student who swaps two neighbouring steps of the six then keeps 2.6 of the 3 marks (14 of the 15 pairs right, one wrong), and with it off earns nothing.

The marking hours against an all-written paper

These are our assumptions, not measurements: three minutes to mark each longer answer, two for the short one, an hour of calibration for each of four markers against two, and an hour on the item statistics. We computed the table in code. This paper needs 15% of the hours. At ten hours a week from each marker, results take three and a half weeks against one. With your own figures, first marking is students times questions times minutes.

What should you read in the item statistics afterwards?

Read each question's difficulty and discrimination before any score is released, and change a question's marking only for a fault found in the question itself.

With 500 students the figures are steady: the upper and lower groups EdProcto compares hold 135 students each. Its results flag a question most of the class missed, one the stronger students got wrong more often than the weaker, and a wrong option that drew more students than the key. Our guide to when to throw a question out weighs re-keying, crediting everyone and dropping.

Applying the decision takes one press in EdProcto for two of the three. For a corrected key, change it in the builder and press "Re-mark this quiz" on the results page, then "Yes, re-mark": every attempt handed in is marked again from the keys as they stand, and the page says how many have new marks. To drop a question, delete it in the builder and re-mark: the paper is then scored out of 48 where a 2-mark question has gone. Crediting everyone is done attempt by attempt, with "Add a note, or set the points by hand" under the answer, so on 500 scripts dropping the question is the quicker fair route.

Until the written answer is graded it counts as zero, so a student with 35 of the 44 rule-made marks shows 70%. Set "Show the score" to "When released".

How does EdProcto mark a paper for 500 students?

EdProcto is the best tool for this job when most marks must come by rule and the paper must not shrink to multiple choice, because every rule-marked shape in the worked paper has its own rule, with part marks where part of an answer is right. Questions reach each device without their keys, and the marks are made on the server at hand-in. To begin, paste the lecture notes into the quiz maker on the home page, and Procto AI drafts questions for you to edit.

Free takes 30 students a quiz, Tutor 100 and Educator 500, and Educator carries the full analysis, reliability included. A class of exactly 500 fills an Educator quiz, with no spare place. The School plan gives each teacher the Educator plan, so one quiz has the same ceiling. Above 500, run two quizzes with the same questions, or ask about the Institution licence, which covers every student and is arranged by a call. Pricing has the plans and the page for schools the licence. Every score is yours to release.

Quick questions

How do you grade 500 students with two markers?

Let a rule mark every answer it can, and read one short written answer each. On our assumptions that takes about 21 marking hours, against about 142 all-written.

Can auto-graded questions go beyond multiple choice?

Yes. Numbers within a tolerance, ordering, matching, sorting, gaps, multiple answers and a point on an image are all marked by rule, as EdProcto's templates show.

Is marking by rule fair to a student who makes one slip?

Not on a multi-step problem with one answer box, where the slip costs every mark. Split it into steps, print each step's starting figure, and set a tolerance.

Which EdProcto plan takes a class of 500?

Educator, whose ceiling is 500 students a quiz, so the class fills it exactly. A larger class is two quizzes or the Institution licence: see the page for universities.

References

• Ashton, H. S., Beevers, C. E., Korabinski, A. A., & Youngson, M. A. (2006). Incorporating partial credit in computer-aided assessment of mathematics in secondary education. British Journal of Educational Technology, 37(1), 93–119. https://doi.org/10.1111/j.1467-8535.2005.00512.x
• Bird, F. L., & Yucel, R. (2013). Improving marking reliability of scientific writing with the Developing Understanding of Assessment for Learning programme. Assessment & Evaluation in Higher Education, 38(5), 536–553. https://doi.org/10.1080/02602938.2012.658155
• Bloxham, S., den Outer, B., Hudson, J., & Price, M. (2016). Let's stop the pretence of consistent marking: Exploring the multiple limitations of assessment criteria. Assessment & Evaluation in Higher Education, 41(3), 466–481. https://doi.org/10.1080/02602938.2015.1024607
• Broadbent, J., Panadero, E., & Boud, D. (2018). Implementing summative assessment with a formative flavour: A case study in a large class. Assessment & Evaluation in Higher Education, 43(2), 307–322. https://doi.org/10.1080/02602938.2017.1343455
• Chen, B., West, M., & Zilles, C. (2018). How much randomization is needed to deter collaborative cheating on asynchronous exams? In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (pp. 1–10). ACM. https://doi.org/10.1145/3231644.3231664
• Denny, P., Luxton-Reilly, A., & Simon, B. (2008). Evaluating a new exam question: Parsons problems. In Proceedings of the Fourth International Workshop on Computing Education Research (pp. 113–124). ACM. https://doi.org/10.1145/1404520.1404532
• Momsen, J. L., Long, T. M., Wyse, S. A., & Ebert-May, D. (2010). Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills. CBE Life Sciences Education, 9(4), 435–440. https://doi.org/10.1187/cbe.10-01-0001
• Rodriguez, M. C. (2003). Construct equivalence of multiple-choice and constructed-response items: A random effects synthesis of correlations. Journal of Educational Measurement, 40(2), 163–184. https://doi.org/10.1111/j.1745-3984.2003.tb01102.x
• Scott, M., Stelzer, T., & Gladding, G. (2006). Evaluating multiple-choice exams in large introductory physics courses. Physical Review Special Topics - Physics Education Research, 2(2), Article 020102. https://doi.org/10.1103/PhysRevSTPER.2.020102
• Stanger-Hall, K. F. (2012). Multiple-choice exams: An obstacle for higher-level thinking in introductory science classes. CBE Life Sciences Education, 11(3), 294–306. https://doi.org/10.1187/cbe.11-11-0100