A question bank is a board's examination kept in stock: every question it could set, each filed with its blueprint cell, its review record and its statistics. Testing texts call it an item bank. To build one, a body with three people in the office needs the same parts as the largest examiners, in smaller quantities.
This guide follows the process in order: blueprint, item writing, review, trial, statistics, reuse, drift and retirement, with the research at each step and a worked bank for 400 candidates a year. Where a figure is our working rule, not a research finding or a standard, we say so.
EdProcto publishes this guide and does two of the steps: it drafts questions from a body's own syllabus and reports each sitting's statistics. Its page for associations and boards sets out the offer, and the last section says what question bank management needs that it does not do.
Where does a question bank start? The blueprint, from a practice analysis
A bank starts with the blueprint, because a question that fits no cell has no place in it. As the American Board for Certification in Orthotics, Prosthetics & Pedorthics (2021) quotes Standard 11.13 of the Standards for Educational and Psychological Testing, the content of a credentialing test should be defined clearly and justified by its importance to credential-worthy performance, and a job or practice analysis typically supplies the main basis.
The National Board for Certification in Occupational Therapy (n.d.) says it repeats its practice analysis every five years: a panel of practitioners, then a survey of thousands of certificants.
Both are reported as theirs: whether any standard or accreditation scheme applies to your body is for you to establish. Our exam blueprint guide has a method for a small one.
How do you run item writing and train the writers?
Give every writer the same written rules, train them first, and commission by blueprint cell. Haladyna et al. (2002) drew 31 guidelines for multiple-choice questions from 27 textbooks and 27 studies; multiple-choice questions that measure covers the main ones. Procto AI, on EdProcto's home page, drafts from the part of the syllabus you paste, for a trained writer to edit.
Training shows. Jozefowicz et al. (2002) had three test developers rate 555 questions from three United States medical schools on a five-point scale, blind to the writers' training. The 92 from writers a national board had trained averaged 4.24, and the other 463 averaged 2.03.
How many written questions survive?
Between a third and three quarters, in the one record we found. Case et al. (2001) followed questions written in 1997 for a medical licensing examination into live use. Standing committees, whose new members had a two-day workshop, wrote 2,356 questions between 48 authors, and 27 per author reached the live pool: about 55%, on our arithmetic from the paper's rounded figures. Task forces writing in pairs beside an editor reached about 77%, and volunteers sent a manual about 34%.
What should review check before a question is trialled?
Review checks three things in three passes, each by someone other than the writer. The wording of the checks is ours.
- Content. Is the key the only defensible answer, and does the question fit its cell and current practice?
- Editorial. Is the stem clear, with no clue in length or grammar?
- Sensitivity and bias. Does it lean on a stereotype, a regional term or knowledge outside the syllabus?
What research says review catches
Committee review raises measured quality. Wallach et al. (2006) had national board staff rate one medical school's questions blind on a five-point scale: 2.51 before the school set guidelines and had a committee read every question aloud, 3.16 in the first year after and 3.59 in the second.
Review misses things too. Engelhard et al. (1990) asked 42 bias review judges on teacher certification tests to predict which questions would behave differently for Black and White examinees; their agreement with the data was generally no better than chance. Over six years at one medical school, Srisomsak et al. (2025) found 81 of 14,238 questions (0.6%) needed a correction after the sitting, mostly a second right answer or a wrong key.
How do you trial a question unscored before it counts?
Seat new questions among the scored ones, unmarked, and bank those whose figures hold. The occupational therapy board's exams carry field-test questions that do not count, analysed once a large sample has seen them. The American Society for Quality (n.d.) publishes the proportion for one exam: 110 questions on screen, 100 scored and 10 unscored.
We found no agreed figure for the responses a trial needs. Young et al. (2017) show the floor: across 22 cohorts of 10 to 15 students, 29 of 32 questions were rated both poor and excellent for discrimination at least once. Joncas et al. (2018) judged that 180 to 200 students gave "some stability". Our working figure is 100.
A body too small to trial can score a new question on first use and hold results until its figures are read: see when to throw a question out after the exam.
Which statistics are kept with each question, and at what thresholds?
For every use of every question keep the date and form, the number of candidates, the difficulty (the share answering correctly), a discrimination figure, the count for each option and the time taken. Beside them sit the fixed facts: blueprint cell, source of the answer, writer, review decisions and version.
The thresholds are conventions, and the published ones disagree.
| Figure | One line | Another |
|---|---|---|
| Discrimination | Under 0.10 poor, 0.10 to 0.19 low, 0.20 and over acceptable or better: the scale applied by Young et al. (2017) | Negative flawed, under 0.15 poor, 0.15 to 0.25 "recommended", 0.25 and over good (Srisomsak et al., 2025) |
| Difficulty | Exclude under 0.2 or over 0.8, a rule of thumb the same study tested | Flag under 0.25 (Srisomsak et al., 2025) |
| EdProcto's results print a sentence | Difficulty under 0.30, or 0.90 and over | Any negative discrimination |
A question at 0.17 is "low" on the first scale and "recommended" on the second. No threshold replaces reading: the screen in Srisomsak et al. (2025) missed 11 of 77 corrected questions. Fix your lines before the sitting, and reread any question that crosses one. Item analysis for teachers explains each figure.
How many questions does a bank need? A worked example
A bank needs enough questions to fill every form in the reuse window without breaking the reuse rule, plus a reserve. Case et al. (2001) name the factors: test length, number of forms, repeat testing and exposure rules. Bergstrom and Gershon (1995) observed that even agencies testing few candidates usually held several hundred questions.
What the research says about reuse
Reused questions grew easier in the one cohort study we read: Joncas et al. (2018) followed 1,629 of them through five cohorts at one medical school. The share answering correctly rose from 79.8% at first use to 85.2% at the fourth, and discrimination fell among questions used three or four times. Sharing between cohorts is one explanation the authors offer; they cannot confirm it.
The bank for 400 candidates a year
Take 400 candidates a year, two sittings of 200, and one form of 100 scored questions at each. Our working rule, drawn from the Joncas result: no question is scored more than twice in any five years. Five years is 10 forms, or 1,000 scored places.
| Scored uses in five years | Questions needed | With a 20% reserve |
|---|---|---|
| 1 | 1,000 | 1,200 |
| 2 | 500 | 600 |
| 3 | 334 | 401 |
The reserve is our margin, so a cell can lose a question without breaking the rule. The target is 600.
The yearly writing target
Each year fills 200 scored places. Cap a question at four scored uses and assume that, with early losses, the average is 2.5: then 80 new questions must reach the bank each year. Assume three in four written questions pass review and two in three of those pass trial, one half overall and inside the Case et al. (2001) range. Then 120 go to trial and 160 are written.
Trial is the bottleneck. Ten unscored questions for each of 400 candidates give 4,000 trial responses a year. At 100 responses a question that is 40 trialled and about 27 banked, a third of the 80. Trialling all 120 would take 30 unscored questions per candidate. So a small body chooses: a longer sitting, fewer responses per trial, a longer life for each question, or scored first use. A new bank also starts looser: 600 questions at 80 a year take seven and a half years.
Why does a question that worked five years ago stop working?
A question stops working because the subject, the candidates or its secrecy has changed. The first is item drift. Bock et al. (1988) followed a physics achievement test for 10 years and found question difficulty drifting in directions tied to content, as school curricula changed emphasis; discrimination did not drift. For a board the plainest case is a revised rulebook: a key can become wrong the day the rule changes.
The last is a leak. Zimmermann et al. (2016) simulated candidates who knew some questions in advance, on an admission test that reuses 40 of its 80 questions each year. The most sensitive signs were a fall in a question's discrimination against its earlier uses and a new dependence between the leaked questions. Groups under 5% of candidates were not reliably detected. The authors' advice is to drop a suspected question from scoring and not reuse it.
Keeping the bank secure
Our working rules: keep the bank in one place with named access, give writers their cells and not the bank, and record every form a question appeared on.
In EdProcto the answer keys stay on the server and the marks are made there, and candidates are told beforehand which integrity measures apply. A quiz can be published as a kit that others open a copy of, keys included: leave that off for an examination. No setting makes cheating impossible.
When should exam questions be retired or rewritten?
Retire a question when its content is no longer true or no longer on the blueprint, when it reaches its use limit, when a leak is suspected, or when its figures fail at two sittings running. The last is our rule. Bergstrom and Gershon (1995) give content experts a scheduled review for three things: questions still current, duplicates flagged, and content still matching the blueprint.
Rewrite when the idea is sound and the fault is local, such as a wrong option nobody chooses. A rewritten question is a new question: in the programme Joncas et al. (2018) studied, any significant change, even rephrasing an option, made it one. It goes back through review and trial, and the old statistics stay with the old version. Keep retired questions on file, with the reason.
How does EdProcto fit a board's question bank?
EdProcto is the best tool for the two ends of this job when the staff is small, because it drafts from the body's own standard and writes each sitting's findings as plain sentences.
With each question it keeps a topic, which can carry the blueprint area; a difficulty label of easy, medium or hard, which is a judgement and not a statistic; an explanation; and, when Procto AI wrote the question from your material, where in the source the answer is.
After a sitting it gives each question's difficulty, its discrimination between upper and lower groups (not under six submissions), its correlation with the rest of the quiz and its median time; for single-answer questions, the takers of each option; for the quiz, a reliability estimate. An option nobody took, among ten or more candidates, draws the line "A wrong answer nobody picks is doing no work; make it plausible."
What a bank needs that it does not do today:
- Questions live inside quizzes. No bank spans quizzes, and no question keeps a history from one sitting to the next.
- There are no unscored trial questions. A question switched to "Not marked" is left out of the score and of the analysis.
- It does not assemble forms to a blueprint. It can shuffle question and option order for each candidate, and on the Educator plan Procto AI can write each candidate a set of their own, which is not a form assembled from your bank.
- It has no export of the statistics and no review workflow with roles.
So the bank, its histories and the reuse rule live in your own spreadsheet, with each sitting's figures copied in.
Free has the core templates, marking by rule and results for each question. Tutor has every template and 50 questions a month from Procto AI. Educator has the full analysis, certificates and 500 candidates a quiz. Pricing lists each plan, and a licence for a whole body starts with a call: see for schools and ask for a demonstration. EdProcto does not make an examination meet any standard or accreditation scheme, and is not affiliated with any board.
Quick questions
How many questions should an item bank have?
Divide the scored places your forms fill in the reuse window by the uses you allow. Two forms a year of 100 questions, each scored at most twice in five years, need 500, or 600 with a reserve.
How often can an exam question be reused?
No standard we read fixes a number. Joncas et al. (2018) saw questions grow easier with each use within five years, so our working rule is twice in five years.
When should exam questions be retired?
When the content is out of date or off the blueprint, the use limit is reached, a leak is suspected, or the figures fail twice.
Can EdProcto hold our question bank?
Not today: its questions live inside quizzes. EdProcto drafts questions from your syllabus, runs the sitting and reports that sitting's statistics, as its page for associations and boards describes.
References
- American Board for Certification in Orthotics, Prosthetics & Pedorthics. (2021). Practice analysis of ABC certified therapeutic shoe fitters. https://www.abcop.org/publication/section/practice-analysis-of-abc-certified-therapeutic-shoe-fitters/introduction
- American Society for Quality. (n.d.). Six Sigma Green Belt certification CSSGB. https://www.asq.org/cert/six-sigma-green-belt
- Bergstrom, B. A., & Gershon, R. C. (1995). Item banking. In J. C. Impara (Ed.), Licensure testing: Purposes, procedures, and practices. Buros Institute of Mental Measurements. https://digitalcommons.unl.edu/buroslicensure/13
- Bock, R. D., Muraki, E., & Pfeiffenberger, W. (1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25(4), 275–285. https://doi.org/10.1111/j.1745-3984.1988.tb00308.x
- Case, S. M., Holtzman, K., & Ripkey, D. R. (2001). Developing an item pool for CBT: A practical comparison of three models of item writing. Academic Medicine, 76(10, Suppl.), S111–S113. https://doi.org/10.1097/00001888-200110001-00037
- Engelhard, G., Jr., Hansche, L., & Rutledge, K. E. (1990). Accuracy of bias review judges in identifying differential item functioning on teacher certification tests. Applied Measurement in Education, 3(4), 347–360. https://doi.org/10.1207/s15324818ame0304_4
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5
- Joncas, S. X., St-Onge, C., Bourque, S., & Farand, P. (2018). Re-using questions in classroom-based assessment: An exploratory study at the undergraduate medical education level. Perspectives on Medical Education, 7(6), 373–378. https://doi.org/10.1007/s40037-018-0482-1
- Jozefowicz, R. F., Koeppen, B. M., Case, S., Galbraith, R., Swanson, D., & Glew, R. H. (2002). The quality of in-house medical school examinations. Academic Medicine, 77(2), 156–161. https://doi.org/10.1097/00001888-200202000-00016
- National Board for Certification in Occupational Therapy. (n.d.). Foundations of the certification examinations. https://www.nbcot.org/exam-info/foundations
- Srisomsak, V., Sitticharoon, C., Keadkraichaiwat, I., Meethes, S., & Inpaen, I. (2025). Detection of flawed multiple-choice questions in preclinical medical education using item difficulty and discrimination indices: A six-year analysis. BMC Medical Education, 26, Article 92. https://doi.org/10.1186/s12909-025-08204-5
- Wallach, P. M., Crespo, L. M., Holtzman, K. Z., Galbraith, R. M., & Swanson, D. B. (2006). Use of a committee review process to improve the quality of course examinations. Advances in Health Sciences Education, 11(1), 61–68. https://doi.org/10.1007/s10459-004-7515-8
- Young, M., Cummings, B.-A., & St-Onge, C. (2017). Ensuring the quality of multiple-choice exams administered to small cohorts: A cautionary tale. Perspectives on Medical Education, 6(1), 21–28. https://doi.org/10.1007/s40037-016-0322-0
- Zimmermann, S., Klusmann, D., & Hampe, W. (2016). Are exam questions known in advance? Using local dependence to detect cheating. PLOS ONE, 11(12), Article e0167545. https://doi.org/10.1371/journal.pone.0167545