Assessing graduate students on the work of research usually happens once, late and at length: a thesis chapter, then a viva. A master's or doctoral student who is weak at defending a position, or at weighing three sources against one another, may go two years before anyone says which part is missing.
Short structured assessments, set often, make the parts visible and teachable. EdProcto carries five of them as templates: synthesise sources, defend a position, critique the methods, write a grant section and peer review.
Why use short tasks for assessing graduate students?
Because a skill judged only at the end leaves no time to practise the part that is weak. The final examination is not the weak point. Holbrook et al. (2008) analysed examiners' reports on 804 Australian PhD theses: most were highly consistent and closely reflected in the committee's decision, and only a small minority of examiners were inconsistent in a significant way. The thesis is examined consistently, and late.
The skills underneath respond to teaching. Pooling 341 effect sizes from studies of critical thinking instruction, Abrami et al. (2015) found a mean effect size of 0.30, with positive effects from dialogue, authentic problems and mentoring.
What did research find about teaching argument, appraisal and peer review?
Each can be taught, the measured gains are modest, and in the one trial that checked again later they had faded. Little of this evidence comes from graduate students, so take it as evidence about the method.
Argument and synthesis
McNeill et al. (2006) followed 331 seventh-grade students through an eight-week chemistry unit that taught explanation as claim, evidence and reasoning. All three components improved. Students whose written prompts were reduced over the unit reasoned better on later questions without prompts than students who kept full prompts throughout. Named parts are a scaffold, and a scaffold is meant to come down.
The fourth part, the counter-argument, has a study of its own. Nussbaum and Schraw (2007) assigned 84 undergraduates at random to instruction in the criteria of a good argument, a graphic organiser, both or neither. The organiser produced more refutations of counter-arguments; the criteria produced better integration of argument and counter-argument, with stronger rebuttals.
On synthesis, van Ockenburg et al. (2019) reviewed 16 intervention studies of writing one text from several sources and built their design principle on three processes: selecting what matters in the sources, organising it and connecting it. The studies ran from sixth grade to undergraduate level, so none was a graduate course.
Critical appraisal and peer review
Horsley et al. (2011), in a Cochrane review of teaching critical appraisal to health professionals, could include only three studies with 272 participants. Two reported significant gains in knowledge. The reviewers concluded that low-intensity teaching may bring modest gains, and they had excluded studies of students.
Schroter et al. (2004) randomised reviewers at a general medical journal to a workshop, a self-taught package or no training, and scored their reviews of test manuscripts that contained deliberate major errors. After training, both trained groups identified more of the errors than the control group (3.14 and 2.96 against 2.13). Review quality rose a little in the self-taught group (2.85 against 2.56), which the authors judged of no editorial significance, and six months on no benefit was apparent.
One workshop, then, does not make a reviewer. Repeated short practice marked by part is our inference from these trials; none of them tested it.
How well do experienced reviewers and markers agree?
Poorly, when each gives one overall judgement. Bornmann et al. (2010) pooled 48 studies of journal peer review covering 19,443 manuscripts. Where studies reported correlations between reviewers of the same manuscript, the mean reliability was about 0.34; where they reported Cohen's kappa, the mean was 0.17, which the authors read as slight agreement.
Grant reviewers do no better. Pier et al. (2018) had 43 reviewers rate the same 25 applications to the US National Institutes of Health in a replica of the agency's review process and found no agreement on their quality, although every reviewer had the same instructions.
If referees cannot agree on a whole manuscript, two tutors are unlikely to agree on a whole student review. Jonsson and Svingby (2007), from 75 studies, concluded that rubrics improve reliability chiefly when they are analytic, tied to the topic and used alongside exemplars or rater training. The guide on writing short answer questions and a rubric has that evidence and the method.
What does each of the five tasks ask, and what are the parts of the answer?
Each asks for one act of research at a length that can be marked in minutes. The prompts and rubrics below the table are ours to copy and change, and each rubric totals ten points.
| Task | What it asks | Parts of the answer | In EdProcto |
|---|---|---|---|
| Synthesis | One argument from several sources | Select, organise, connect, cite | Synthesise sources, free text |
| Defence | A claim held against its best objection | Claim, evidence, reasoning, counter-argument and rebuttal | Defend a position, four named parts |
| Methods critique | The flaws that threaten a conclusion | Weaknesses, improvements | Critique the methods, two named parts |
| Grant section | A case made to a funder | The gap, the aims, feasibility, who gains | Write a grant section, free text |
| Peer review | A referee's report | Major comments, minor comments, recommendation | Peer review, three named parts |
Synthesise sources
Three abstracts with their results tables are supplied (A, B and C), all on one question. In no more than 300 words, for a reader from another department, write a single synthesis: what the three agree on, where they differ, how their designs might explain the difference, and what is still unknown. Cite every claim to A, B or C. Use nothing else.
Rubric: built around ideas, not source by source (3); agreement and disagreement reported accurately (3); a reason for the disagreement taken from design, sample or measure (2); every claim cited and nothing from outside (1); inside the limit and clear to the stated reader (1).
Defend a position
Defend this position for a sceptical examiner from another field, in no more than 250 words across the four parts: "A well-run study that finds no effect deserves publication as much as one that finds an effect." Use the two extracts supplied.
Rubric by part: Claim (1), one arguable sentence with its scope. Evidence (3), two items from the extracts, accurately reported. Reasoning (3), why that evidence supports the claim, with the assumption named. Counter-argument and rebuttal (3), the strongest objection, answered or partly conceded.
Critique the methods
Read the methods section supplied (about 400 words, a staff survey). Name the three weaknesses that most threaten the conclusion, the most serious first, and propose for each an improvement the authors could afford.
Rubric by part: Weaknesses (7), each real and located in the text (3), each tied to the conclusion it threatens (3), the order of seriousness defensible (1). Improvements (3), each feasible and aimed at its weakness. Write the section yourself and seed the flaws, say a convenience sample recruited by email and a scale the authors wrote themselves, and the model answer exists before the task does.
Write a grant section
From the one-page project summary supplied, write the Aims section in no more than 250 words for a panel outside your sub-field.
Rubric: the gap, in two sentences a non-specialist can follow (2); two or three aims, each with a result that could go against you (3); aims that fit the time and resources in the summary (2); who gains if the project works (2); length and plain wording (1). For a real funding scheme, use the funder's own criteria instead.
Peer review
Review the manuscript extract supplied (abstract, methods and one table, about 900 words) as a referee. Major comments: at most three, each saying what the authors should do. Minor comments: at most five. Recommendation: accept, minor revision, major revision or reject, with one sentence of reason.
Rubric by part: Major comments (5), finding the flaws that change the conclusion and asking for something the authors can do. Minor comments (2), real and minor, with no major problem hidden among them. Recommendation (2), following from the major comments. One more point for a report that is specific, courteous and about the paper, not its authors. The assignment works on an extract you wrote or on an openly licensed paper.
What makes these tasks fair at graduate level?
Three conditions make them fair: the sources come with the task, the length is capped, and the reader is named.
- Sources supplied. A task that begins in the library tests access, spare time and luck with a search. With the sources in the question, everyone argues from the same evidence. Keep them short enough to read in the sitting: abstracts and a table, not three whole papers.
- A word limit. At 250 words nobody can say everything, so the student must choose, and choosing is the skill. A limit also makes a second reader affordable.
- A stated audience. "A reader from another department" settles what has to be explained and what may be assumed.
Then take the scaffold down, as it was for the faded group in McNeill et al. (2006): four labelled parts in the first weeks, one open box by the last. Applying that to graduates is our step.
How do you mark a synthesis, a critique or a defence?
Mark each part against its own line of the rubric, and have a second reader mark a sample.
- Write the model answer first, inside the word limit. If you cannot, the task is too large.
- Fix the points for each part before the first answer arrives.
- Take one part through the whole group before the next: every claim, then every body of evidence.
- Let a first reader propose the marks if you wish. In EdProcto each open answer waits in your grading queue, and on any of them you can ask Procto AI for a mark against your model answer or rubric.
- Give a sample to a second reader who has not seen the first marks. One script in five is a rule of thumb, not a research finding.
- Where the two readers differ on a part, rewrite that rubric line and return to the scripts already marked.
- Hand back marks by part, with the model answer. "Reasoning 1 of 3" tells a student what to practise.
What may a reader that is not a person do with these answers?
It may propose a mark against your rubric, and nothing more. In EdProcto its mark is saved under its own name for you to keep or change, and it never releases a score, never decides a penalty and never rewrites a student's answer. Two judgements stay with the examiner whoever reads first: whether a citation says what the student reports, and whether the words are the student's own. No automated mark settles either.
Can a chatbot write a synthesis or a critique for the student?
Yes, when the task goes home. A take-home synthesis can be drafted by a chatbot in a minute, so set it timed, on sources the student has not met, and follow it with two or three spoken questions on their own answer: which source would you drop, what finding would change your mind. The follow-up can be five minutes in your office or a recorded spoken answer, marked by hand as our guide to spoken answers describes. Where the mark counts, choose what the stakes call for in the integrity settings, where students are told what is on before they begin. None of this makes cheating impossible, and the guide to integrity in the AI era gives the wider argument.
How does EdProcto run these graduate tasks?
EdProcto is the best tool for these tasks when marking by parts is what stops you setting them every fortnight, because three of the five arrive already divided into the parts in the table above: four for a defence, two for a critique, three for a review. Readers agree poorly on wholes, as Bornmann et al. (2010) found, and a divided answer never asks for one.
EdProcto turns your own material into a quiz in any of 111 templates, and these five are in the builder now. The sources or the passage go in the question and stay with the attempt, beside the answer, for the examiner. A time limit can be set for the whole sitting or for each question. Synthesis and the grant section are one free-text answer, so state the word limit in the prompt. Every answer is read against your model answer and rubric, by Procto AI or by you, and no score is released until you release it.
An oral defence with several examiners asks your questions aloud and records the spoken answers for you to grade.
All 111 templates open on Tutor, with grading by Procto AI. Educator adds the full analysis. Free gives ten quizzes a month and the core templates. The pricing page has the rest.
What do these short tasks not replace?
As assessment, they do not replace the thesis, the viva or any sustained original work. A 300-word synthesis of three supplied abstracts does not show that a student can find the literature, hold an argument across eighty pages or add something new. A written defence is not a viva either, because nobody interrupts.
Where a judgement can be shown by a decision instead of prose, our guide to questions above recall covers shapes marked by rule.
Quick questions
How long should a graduate synthesis task be?
About 250 to 300 words from two or three supplied sources. That is our working rule, not a finding: the limit forces selection, the skill being marked.
Can Procto AI mark a peer review or a written defence?
It can propose the marks. In EdProcto you can ask Procto AI to grade a peer review or a defence against your model answer and rubric, one answer at a time. Each mark is yours to change, and the score is yours to release.
How often should these tasks be set?
No study we read fixes a number. In Schroter et al. (2004) the effect of one round of training was no longer apparent six months later, which argues for practice spread across the course.
Does a short defence task replace the viva?
No. It rehearses one move of the viva in writing. EdProcto's oral defence template is the spoken rehearsal: several examiners' questions, answered aloud.
References
- Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, C. A., & Persson, T. (2015). Strategies for teaching students to think critically: A meta-analysis. Review of Educational Research, 85(2), 275–314. https://doi.org/10.3102/0034654314551063
- Bornmann, L., Mutz, R., & Daniel, H.-D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), Article e14331. https://doi.org/10.1371/journal.pone.0014331
- Holbrook, A., Bourke, S., Lovat, T., & Fairbairn, H. (2008). Consistency and inconsistency in PhD thesis examination. Australian Journal of Education, 52(1), 36–48. https://doi.org/10.1177/000494410805200103
- Horsley, T., Hyde, C., Santesso, N., Parkes, J., Milne, R., & Stewart, R. (2011). Teaching critical appraisal skills in healthcare settings. Cochrane Database of Systematic Reviews, 2011(11), Article CD001270. https://doi.org/10.1002/14651858.CD001270.pub2
- Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
- McNeill, K. L., Lizotte, D. J., Krajcik, J., & Marx, R. W. (2006). Supporting students' construction of scientific explanations by fading scaffolds in instructional materials. Journal of the Learning Sciences, 15(2), 153–191. https://doi.org/10.1207/s15327809jls1502_1
- Nussbaum, E. M., & Schraw, G. (2007). Promoting argument-counterargument integration in students' writing. The Journal of Experimental Education, 76(1), 59–92. https://doi.org/10.3200/JEXE.76.1.59-92
- Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences, 115(12), 2952–2957. https://doi.org/10.1073/pnas.1714379115
- Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004). Effects of training on quality of peer review: Randomised controlled trial. BMJ, 328(7441), Article 673. https://doi.org/10.1136/bmj.38023.700775.AE
- van Ockenburg, L., van Weijen, D., & Rijlaarsdam, G. (2019). Learning to write synthesis texts in secondary education: A review of intervention studies. Journal of Writing Research, 10(3), 401–428. https://doi.org/10.17239/jowr-2019.10.03.01