A quiz comes back with a tidy average, and the scores go into the gradebook as "knows unit three". Validity is the question of whether they can bear that reading.
The figures inside a quiz say how its questions behaved. Whether the right things were asked, in a way that let students show them, is a judgement that needs the syllabus, a colleague, a second measure and a few students.
EdProcto, which publishes this guide, shows evidence that bears on that judgement in a quiz's results. The judgement belongs to the lecturer, trainer or head of department.
What is validity, and does it belong to the quiz?
Validity is how far the evidence supports the meaning you give a score and the use you make of it, so it belongs to that use and not to the quiz. Cronbach and Meehl (1955) put it so: "One does not validate a test, but only a principle for making inferences." Kane (2013) makes it his first point: what is validated is a proposed interpretation or use of scores, never the test. Validation, for him, is an argument: list the claims that lead from the answers to the use, and judge each.
Cook and Hatala (2016) add three points. Validity comes in degrees, not as a yes or a no. It is never proved, only supported or undermined as evidence gathers. And "validated" on a test says that someone once looked, not what they found.
One quiz, two uses
Ten questions on last week's lecture can safely choose what to go over on Monday. Deciding who goes on placement asks far more of them: the more ambitious the claim, Kane (2013) holds, the more support it needs.
How did one validity replace content, criterion and construct validity?
The three became kinds of evidence for one question, in three steps.
- 1954. A committee of the American Psychological Association named four types: predictive, concurrent, content and construct validity. Cronbach and Meehl (1955), two of its members, treat the first two together as criterion-oriented, which leaves the classical three.
- 1989 to 1994. Cook and Hatala (2016) date the first modern framework to Messick in 1989. Messick (1994) argued that three separate, interchangeable types made a broken-up view that left out the social consequences of using a score, and folded content, criteria and consequences into one construct validity.
- 1999 and 2014. The Standards for Educational and Psychological Testing adopted that framework, with five sources of evidence (Cook and Hatala, 2016).
On 10 October 2026 we worked from the full text of Cronbach and Meehl (1955), Cook and Hatala (2016) and Gernsbacher et al. (2020), and from the abstracts of the other sources. The Standards, published as a file we did not open, appear as Cook and Hatala report them.
| Older name | Evidence today | Check below |
|---|---|---|
| Content validity | Content: the questions sample what was taught | 1 and 2 |
| Criterion validity | Relations with other measures | 3 |
| Construct validity | Now the whole; also internal structure, the item figures in EdProcto's results | 4 |
| None | Response processes (how answers were reached) and consequences (what the score led to) | 5; after the quiz |
What are the two threats to a quiz's validity?
There are two: the quiz leaves out part of what was taught, or the score moves for reasons that are not the subject. Downing and Haladyna (2004) give them their names in measurement: construct under-representation and construct-irrelevant variance. Our guide to exam blueprints cites their essay for the first; it is here for the second.
The quiz leaves out part of what was taught
Six weeks are taught and the quiz is written from the last two, so "knows weeks five and six" is filed as "knows the unit". It happens by level too: a unit taught as problem solving and tested as definitions, the subject of questions above recall.
The score moves for reasons that are not the subject
- Reading load. Downing and Haladyna (2004) name questions written at the wrong reading level. When Bosher and Bowles (2008) reworded pathophysiology exam questions to cut the reading and keep the content, students reading in a second language understood them better in most cases.
- Speed. A limit that stops students considering every question makes pace part of the score. In a 1928 study that Gernsbacher et al. (2020) report, of nearly 200 students over nearly 15 exams each, time taken and score were barely related: pace is not knowledge.
- Trick wording and test-wiseness. Tarrant and Ware (2008) had a panel review ten nursing papers: 47.3% of the questions were flawed. On the sound questions alone, fewer students passed (90.6% against 94.3%) and more reached 80% (20.9% against 14.5%), so flaws had helped borderline students and cost the strongest marks.
- The device. Dadey et al. (2018) reviewed how touch screens, keyboards and screen size relate to performance, and hold that scores from different devices need evidence that they compare.
How do you gather validity evidence without a psychometrician?
With five checks, two before the quiz and three after it.
- Set topics against questions, and both against teaching time. List each topic with the hours it was taught and the marks it gets. Cook and Hatala (2016) list a test blueprint and an expert panel among the evidence on content. The check against hours is our working rule: a third of the teaching and a tenth of the marks needs a reason. In EdProcto each question has a box "Topic (for the results by topic)", and "By topic" in the results prints the number of questions beside each topic.
- Have a colleague read each question for what it really asks. Hand over the questions without their topics and ask of each: what must a student know or do to get this right? "Spot the word 'always'" or "read 200 words quickly" is a finding.
- Compare the scores with another measure of the same students: coursework, a practical, last term's exam. Cook and Hatala (2016) count agreement with a measure of the same thing, and its absence with a measure of something else, as evidence. "Download the marks" in EdProcto gives a spreadsheet row for each sitting.
- Look at who the hard questions were hard for. They should be hard for the students who are weak on the rest, which "Separates" reports inside the quiz. Then split the class by something that should not matter, such as who used the whole time or who sat on a phone, and compare question by question.
- Ask a few students to think aloud. Three or four talk through five questions with no clock. Listen for a right answer reached by the wrong route and a wrong one caused by the wording. When Sato et al. (2019) set 22 undergraduates' exam-style answers beside their written-out thinking, mark and understanding disagreed in over a quarter of the data.
In a small class the figures in checks 3 and 4 are rough, as item analysis shows.
How does validity differ from reliability?
Reliability asks whether the quiz would put the same students in the same order again; validity asks whether that order means what you say. A quiz can do the first without the second: on Cook and Hatala's (2016) account, plenty of evidence of reliability does not remove the need for evidence about content, other measures or consequences.
A time limit shows how. Gernsbacher et al. (2020) pass on a textbook's warning that the reliability of a timed test is inflated, because students keep a steady pace and the figure counts that as consistency. The example below does exactly this, and test reliability explains the figure.
What do the consequences of a score have to do with validity?
They are part of it: a use of a score is judged by what it leads to as well as by what the score means. Messick (1994) put the social consequences of score use inside validity, and Kane (2013) holds that negative consequences can make a use unacceptable while the reading of the score still stands. Cook and Lineberry (2016) note that this evidence has no counterpart among the three older types, and that reviews in health professions education found it in only 5% to 20% of studies.
For a quiz, ask what the score did. Did the students it sent to a revision group catch up? Did a line at 60% turn away people who went on to do well (see setting a pass mark)?
A worked example: a twenty-question quiz that measured reading speed
This class of twelve is invented. A four-week statistics unit gives a week to sampling, a week to confidence intervals and two to hypothesis tests. Its quiz has twenty one-mark questions and a twenty-minute limit: ten one-line questions on the first two topics, then ten on hypothesis tests that each open with a case of about 150 words. Half the teaching, half the marks: check 1 passes.
EdProcto's analysis code, run on the twelve sittings, returned an average of 59.6%, eight of twelve passing at 60%, a reliability of 0.81 and two sentences: "Reteach before moving on: Hypothesis tests (50.8%)." and "The quiz measured consistently (reliability 0.8 or above): the order of the students can be trusted, though one close to the pass mark could fall either side."
What in the same report argued against reteaching
- "Median time" was the full twenty minutes.
- "Median t" was 36 seconds on each of the first ten questions and 116 on Q11 to Q14, with the note "Q11 took more than three times as long as a typical question. Either it is much harder or its wording is unclear."
- "Got it" fell from 83% on Q11 to 25% on Q19, while "Separates" stayed at 0.67 or more on each of the last six: inside the quiz, nothing looked wrong.
Then two sums of ours, beside the downloaded marks and a coursework mark earned without a clock.
| Q1 to Q10 | Q11 to Q20 | |
|---|---|---|
| Correlation with the coursework mark | 0.91 | 0.23 |
| Score of the four who finished inside the time | 72.5% | 85% |
| Score of the eight who used all twenty minutes | 66.3% | 33.8% |
The two groups' coursework averaged 68.5 and 65.8, so the groups were much alike, and of the 84 case questions anyone answered, 61 were right. Two of the eight, thinking aloud with no clock, then get Q16 and Q19 right. The last ten questions measured how fast a student reads a case.
The consequences were real: two students with eight of the first ten right failed on 50% and 55%, and two with five of the first ten passed on 70% and 65%. And the reliability flattered. Run on the first ten questions alone, the code returned 0.75; ten rushed questions had raised it to 0.81.
What was changed
Nothing is retaught on this evidence, and no decision rests on the last ten questions. For the next class, copy the quiz with "Duplicate this quiz", cut each case to about 60 words with two questions hanging on it, spread the case questions through the quiz so that topic and position no longer go together, and double the limit after sitting it yourself. The same figures test the change: a median time well inside the limit, and "Got it" no longer falling towards the end.
A checklist before the quiz and one after it
Before:
- The use in a sentence: what will this score decide?
- Checks 1 and 2: the table of topics, hours and marks, and a colleague's read.
- On each question, its topic and the "Difficulty" you expect: "Easy", "Medium" or "Hard".
- Stems cut to what the question needs, and the quiz sat by you before the time is fixed (the unit test online has the evidence on limits).
After:
- The median time against the limit, and "Got it" from the first question to the last.
- "Got it" against the difficulty you expected: an "Easy" question most students missed measured something else.
- Checks 3 to 5: another measure, the hard questions by group, students thinking aloud.
- A note of what the score was used for, and what came of it.
What does EdProcto show that bears on validity?
EdProcto is the best tool for gathering this evidence on a quiz you have run, because one page holds the coverage, the timings, each question's behaviour and the marks to take elsewhere. It shows evidence; the judgement stays with you.
- Coverage. In the builder each question has "Topic (for the results by topic)" and "Difficulty". "By topic" lists each topic you named with its number of questions and the class's score.
- Timings. "Median time" stands at the top beside the limit, "Median t" beside each question when questions come one to a page, and "Time" beside each student.
- Each question. "Got it" is the share of the points the class earned, and "Separates" the top 27% of the class less the bottom 27%, from six scored sittings. "Reliability" is given from six too, and a sentence on it from eight.
- Another measure. "Download the marks" gives a row for each sitting, with "Score %", "Minutes" and the points on each question.
- After you decide. Correct a key or take a question out in the builder and press "Re-mark this quiz": every attempt handed in "is marked again from the questions and keys as they stand now".
Every figure is read on each student's first full sitting. "Got it", the timings, the students' rows and "Download the marks" are on every plan. "By topic", "Separates", "Reliability", "What it says about the teaching" and "What the numbers say" are the full analysis, on the Educator plan. Free takes thirty students a quiz, Tutor 100 and Educator 500: see pricing.
Quick questions
What is validity in simple terms?
Validity is how well the evidence supports what you take a score to mean and what you do with it. One quiz can be sound for one purpose and weak for another.
Can a quiz be reliable but not valid?
Yes. In the worked example the reliability was 0.81, and half the marks followed reading speed.
Are content, criterion and construct validity still used?
As names for kinds of evidence, yes. They are now parts of one validity, joined by evidence on response processes and on consequences.
Does EdProcto tell me whether my quiz is valid?
No tool can. EdProcto's results show the evidence: the score by topic, the timings, each question's behaviour and the marks to compare. The full analysis is on the Educator plan (pricing).
References
- Bosher, S., & Bowles, M. (2008). The effects of linguistic modification on ESL students' comprehension of nursing course test items. Nursing Education Perspectives, 29(3), 165–172. https://pubmed.ncbi.nlm.nih.gov/18575241/
- Cook, D. A., & Hatala, R. (2016). Validation of educational assessments: A primer for simulation and beyond. Advances in Simulation, 1, Article 31. https://doi.org/10.1186/s41077-016-0033-y
- Cook, D. A., & Lineberry, M. (2016). Consequences validity evidence: Evaluating the impact of educational assessments. Academic Medicine, 91(6), 785–795. https://doi.org/10.1097/ACM.0000000000001114
- Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://psychclassics.yorku.ca/Cronbach/construct.htm
- Dadey, N., Lyons, S., & DePascale, C. (2018). The comparability of scores from different digital devices: A literature review and synthesis with recommendations for practice. Applied Measurement in Education, 31(1), 30–50. https://doi.org/10.1080/08957347.2017.1391262
- Downing, S. M., & Haladyna, T. M. (2004). Validity threats: Overcoming interference with proposed interpretations of assessment data. Medical Education, 38(3), 327–333. https://doi.org/10.1046/j.1365-2923.2004.01777.x
- Gernsbacher, M. A., Soicher, R. N., & Becker-Blease, K. A. (2020). Four empirically based reasons not to administer time-limited tests. Translational Issues in Psychological Science, 6(2), 175–190. https://pmc.ncbi.nlm.nih.gov/articles/PMC7314377/
- Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
- Messick, S. (1994). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. ETS Research Report Series, 1994(2). https://doi.org/10.1002/j.2333-8504.1994.tb01618.x
- Sato, B. K., Hill, C. F. C., & Lo, S. M. (2019). Testing the test: Are exams measuring understanding? Biochemistry and Molecular Biology Education, 47(3), 296–302. https://doi.org/10.1002/bmb.21231
- Tarrant, M., & Ware, J. (2008). Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Medical Education, 42(2), 198–206. https://doi.org/10.1111/j.1365-2923.2007.02957.x