Beside each question on a results page stands a percentage headed "Got it". That is item difficulty: the share of the class who got the question right. Test theory calls it the p-value, a name that invites confusion.
EdProcto, which publishes this guide, shows the figure for every question in its results, on every plan. The item analysis guide introduces every figure on that page, and how to set the difficulty of a quiz covers choosing a level beforehand. This guide stays with the one figure, read after the quiz.
The short version: no figure is good by itself. It depends on the quiz's purpose, the number of options and the size of the class.
What is item difficulty, and why is it called a p-value?
Item difficulty is the proportion of students who answered a question correctly, written as 0.72 or 72%: the p-value is that proportion. A higher figure means an easier question, so some systems use a plainer name: Moodle (2026) calls it the facility index.
It is not the p-value of a significance test, which is a probability worked out on the assumption that only chance is at work. This one is a plain share of a class. Tavakol and O'Brien (2022), read in full, warn that it "should not be confused with the p-value related to statistical hypothesis testing".
The figure belongs to the question and the class together
A question has no difficulty by itself. Tavakol and O'Brien (2022) pass on advice credited to Ebel and Frisbie's textbook, which we have not read: say that when this test was given to this group, its difficulty was 63%. A question at 80% last year and 55% this year has not changed. The class or the teaching has.
A question with part marks
Where a question carries part marks, difficulty is the average share of its points that students earned: the University of Washington (n.d.), in a line our other guides do not use, defines it as the average score over the most points on offer. So 64% on a three-gap question does not mean 64% of the class got it right: in the worked example below, eight students in 28 filled every gap.
What is a good item difficulty, according to published guidance?
No single figure is good: the published bands disagree, and each was written for a test meant to rank. Every source below was read on 10 October 2026. The two textbooks they credit were not.
| Source | Band | Where it comes from |
|---|---|---|
| Tavakol and O'Brien (2022) | 0.25 to 0.75 is good | Stated, not derived |
| Hingorjo and Jaleel (2012) | 30 to 70% recommended | Credited to a 2009 textbook |
| Young et al. (2017) | Drop under 0.20 or over 0.80 | A "rule of thumb" from a 2014 textbook |
| University of Washington (n.d.) | Hard to 50%, moderate 51 to 84%, easy from 85% | Report labels the page itself calls arbitrary |
| Rudolph et al. (2019) | Extreme under 0.30 or over 0.90 | Advice to pharmacy lecturers |
| EdProcto | A sentence under 30% and from 90% | Our convention, at the ends Rudolph et al. name |
Take a question at 78%. It is too easy by the first two rows, acceptable by the third, moderate by the fourth and unremarkable by the fifth.
So the bands are conventions: nothing we read derives one, and we found none worked out for a school classroom. All five sources come from health-professions education or a university testing office. They agree only at the ends: under about 0.30 or over about 0.90, look at the question.
Why does the ideal middle sit above 50% on a multiple-choice question?
Because a guess lifts the floor. A student who knows nothing still has one chance in four on a four-option question, so the figure's practical lower bound is 0.25, or 0.33 with three options, a point Rudolph et al. (2019) report from Haladyna and Downing.
A question tells students apart best about midway between that floor and 100%. By our arithmetic the midpoint is 75% with two options, 67% with three, 62.5% with four and 60% with five, and 50% for a typed answer. The figures the University of Washington (n.d.) prints from Lord's 1952 paper sit about ten points higher: that table is in how to set the difficulty of a quiz, and how many options a question should have covers what a guess is worth.
So measure "Got it" from the floor. A true or false question at 61% is eleven points above a coin toss; a four-option question at 61% is thirty-six above its floor.
Does a good figure depend on what the quiz is for?
Yes: a mastery check should be easy, and a test that ranks wants the middle. A mastery check asks whether everyone has the essentials, so high figures mean it worked.
Examinations in professional courses can sit high. Rush et al. (2016), in a finding our other guides do not use, report a mean of 83.3% correct across 1,925 questions set for third-year veterinary students, which the authors call consistent with target values in professional education. Hold such a paper to a ranking band and it is condemned for doing its job.
How much does item difficulty move with class size?
Enough to carry a question across most of those bands. One answer moves the figure by 6.7 points in a class of 15, by 3.3 in 30 and by one in 100.
The table is our arithmetic, not a finding. Suppose a question has a long-run share right among students like yours, each answering independently. Each cell is the range that holds one class's figure in at least 19 classes in 20 (exact binomial sums).
| Long-run share right | Class of 15 | Class of 30 | Class of 100 |
|---|---|---|---|
| 50% | 27 to 73% | 33 to 67% | 40 to 60% |
| 70% | 47 to 93% | 53 to 87% | 61 to 79% |
| 90% | 73 to 100% | 77 to 100% | 84 to 95% |
A real class shares a teacher and a lesson, so real figures move more. Young et al. (2017), in a finding our discrimination guide does not use, followed 32 questions reused with 22 cohorts of 10 to 15 medical students. The 0.20 to 0.80 rule would have removed one question for one cohort and 22 for another.
So treat a figure within two answers of a line as on the line, and pool classes that sit the same paper: see common assessments.
What can item difficulty not tell you by itself?
It cannot tell a hard question from a broken one, or an easy question from a lesson that worked.
At the low end. A question at 20% may be hard, untaught or marked against the wrong key. Rudolph et al. (2019), read in full, work the case: at 0.30 with a discrimination of −0.3 a question is suspect, since the weaker students did better; at 0.30 with 0.5 it is a hard question doing its job. So read discrimination beside it, "Separates" in EdProcto: see the discrimination index and the question with the wrong key.
At the high end. A question at 95% may be too easy, or the class may have learned it: Tavakol and O'Brien (2022) note that the figure cannot say which. It can still be worth asking. Rudolph et al. (2019) write that most students may answer a much-stressed key concept correctly, and that this can be desirable.
Does a low figure mean the question asked for harder thinking?
No. Difficulty is not the level of thinking, and studies disagree on whether the two travel together. Hamamoto Filho et al. (2020), read in full, sorted 119 questions sat by 771 sixth-year medical students into three levels of Bloom's taxonomy: difficulty did not differ significantly between the levels. Rush et al. (2016) found the opposite, the share correct falling as questions rose from recall to analysis.
So judge the level by reading the question: see questions above recall.
What should you do with a question at each end?
Read the question, then match the action to what stands beside the figure.
| "Got it" | Beside it | What to do |
|---|---|---|
| 90% or more | A fact everyone must know | Keep it, early in the quiz |
| 90% or more | A test meant to rank | Rebuild the wrong options: distractors that work |
| Under 30% | Strong students right | Reteach, and keep the question |
| Under 30% | Strong students wrong | Answer it yourself, correct the key, then press "Re-mark this quiz" |
| Under 30% | Answers spread evenly | Teach it, then decide whether it counts |
Hingorjo and Jaleel (2012) advise the same: open with very easy questions or remove them, and review difficult ones for confusing language, controversy or an incorrect key.
How do you read the column in five minutes?
Work down "Got it" once, in this order.
- Check that nothing is waiting (30 seconds). In EdProcto's results, a question with answers still to grade says how many, and its figure rests on the graded ones.
- Find the ends (1 minute). Switch "Each question" to "Heat map" and note every question at 90% or more or under 30%.
- Measure from the floor (30 seconds). Set each choice question's figure against one in two, three, four or five.
- Take the low end first (2 minutes). Answer each low question yourself, without the key. Then read "Separates" on the Educator plan, or open your three highest scorers under "Students" on Free or Tutor.
- Then the high end (30 seconds). Ask of each whether it was meant to be easy.
- Decide (30 seconds). Write one word beside each flagged question: keep, reteach, re-key or rewrite.
A worked example: 28 students, eight questions
This class is invented, and every figure and sentence is what EdProcto's analysis code returned for its answers. Twenty-eight students sat eight questions on the heart and circulation.
| Question, by topic | Shape | "Got it" | "Separates" |
|---|---|---|---|
| Q1 Arteries | Three options | 93% | 0.25 |
| Q2 The left ventricle | Four options | 71% | 0.71 |
| Q3 The pulmonary vein | Four options | 21% | 0.63 |
| Q4 The heart's valves | Four options | 21% | −0.38 |
| Q5 The route of the blood | Three gaps | 64% | 0.67 |
| Q6 Veins and oxygen | True or false | 61% | 0.54 |
| Q7 The thicker wall | Open, four points | 50% | 0.67 |
| Q8 Haemoglobin | One word | 54% | 0.83 |
- Q1 drew "Nearly everyone got Q1, so it says little about who knows more. Fine for confidence, not for measurement." Keep it as the opener.
- Q3 and Q4 share a figure, no better than a blind guess among four options earns. On Q3 the stronger students were right and sixteen chose "Pulmonary artery": a shared wrong idea. On Q4 "Separates" is negative: the twenty who chose "The valves" averaged 62.7% on the quiz, against 41% for the six on the key, "The septum". The key was a slip.
- Q7 read 55% from twenty graded answers while eight waited, and 50% once all were in.
With the key corrected to "The valves" and every attempt marked again, the same code gives Q4 71% with "Separates" at 0.75, and fifteen students pass at 60% instead of twelve. One figure, 21%, had two opposite meanings, and only the column beside it told them apart.
How does EdProcto show item difficulty?
EdProcto is the best tool for reading item difficulty when the scores go out tomorrow. It works the figure out for every question on every plan, counts part marks, and on the Educator plan writes a sentence at each end.
The arithmetic. EdProcto reads each student's first full sitting that was handed in: your own preview, a retry of the misses and a later sitting by the same student are left out. For each question it averages the share of the points each student earned, so part marks count. A blank is marked zero and counts. An answer waiting for a grade stays out of its question's figure until it has one. The figure is rounded to two decimal places and shown as a whole percentage. After a key is corrected, "Re-mark this quiz" marks every attempt handed in again, and the page reads its figures from the new marks.
The page. Under "Each question", a table sets "Got it" and "Median t" beside every question, with "To grade" below the Educator plan and "Separates" and "What the numbers say" on it. The figure changes colour at 50% and at 70%, and the bar beside it under 30% and at 90% or more. "Heat map" shows a tile for each question in three bands, "under 50% got it", "50 to 70%" and "over 70%", with exactly 70% in the top band.
The sentences. In the full analysis, once three answers to a question have a mark, a printed figure of 0.90 or more draws the sentence quoted for Q1, and one under 0.30 draws:
Most of the class missed Q3. Check the key first, then whether this was taught.
A fresh set. With "A fresh set for each student" on, no two students met the same question, so none has a figure: in the table's place the page explains that "no single question was put to the class".
The label in the builder. Each question also carries "Difficulty": "Easy", "Medium" or "Hard". That is a forecast, set before the quiz by you or by Procto AI on a draft, and the analysis neither reads nor changes it.
The plans. Free takes thirty students a quiz and Tutor 100. Educator takes 500 and adds the full analysis. See pricing, then start from the templates or the home page.
Quick questions
What is a good p-value for a test question?
There is no single one. For a test that ranks, published bands run from 0.25 to 0.75 (Tavakol and O'Brien, 2022) or from 30 to 70% (Hingorjo and Jaleel, 2012); a mastery check should sit higher.
Is the item p-value the same as the p-value in statistics?
No. It is the proportion of students who answered correctly, and has nothing to do with the probability a significance test reports.
What is the facility index?
Item difficulty under another name: in Moodle's (2026) definition, the average mark on a question as a percentage.
Where does EdProcto show item difficulty?
In a quiz's results, under "Each question", as "Got it", on every plan. The sentences are in the full analysis on the Educator plan: see pricing.
References
- Hamamoto Filho, P. T., Silva, E., Ribeiro, Z. M. T., Hafner, M. L. M. B., Cecilio-Fernandes, D., & Bicudo, A. M. (2020). Relationships between Bloom's taxonomy, judges' estimation of item difficulty and psychometric properties of items from a progress test: A prospective observational study. Sao Paulo Medical Journal, 138(1), 33–39. https://doi.org/10.1590/1516-3180.2019.0459.R1.19112019
- Hingorjo, M. R., & Jaleel, F. (2012). Analysis of one-best MCQs: The difficulty index, discrimination index and distractor efficiency. Journal of the Pakistan Medical Association, 62(2), 142–147. https://archive.jpma.org.pk/article-details/3255
- Moodle. (2026). Quiz statistics report. MoodleDocs. https://docs.moodle.org/en/Quiz_statistics_report
- Rudolph, M. J., Daugherty, K. K., Ray, M. E., Shuford, V. P., Lebovitz, L., & DiVall, M. V. (2019). Best practices related to examination item construction and post-hoc review. American Journal of Pharmaceutical Education, 83(7), Article 7204. https://doi.org/10.5688/ajpe7204
- Rush, B. R., Rankin, D. C., & White, B. J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16, Article 250. https://doi.org/10.1186/s12909-016-0773-3
- Tavakol, M., & O'Brien, D. (2022). Psychometrics for physicians: Everything a clinician needs to know about assessments in medical education. International Journal of Medical Education, 13, 100–106. https://doi.org/10.5116/ijme.625f.bfb1
- University of Washington. (n.d.). Understanding item analyses. Institutional Assessment & Evaluation. https://www.washington.edu/assessment/scanning-scoring/scoring/reports/item-analysis/
- Young, M., Cummings, B.-A., & St-Onge, C. (2017). Ensuring the quality of multiple-choice exams administered to small cohorts: A cautionary tale. Perspectives on Medical Education, 6(1), 21–28. https://doi.org/10.1007/s40037-016-0322-0