# Reasoning tests for hiring, made fresh for every candidate

**How do I give every job candidate a different reasoning test and still compare their scores?** A reasoning test made fresh for every candidate deals each person their own questions from one rule, so an answer passed on seldom fits the next set. The scores compare as throws of the same dice. On ten questions, two equally able candidates land three or more marks apart about one time in four, so use the score as a floor, not a ranking.

Published 2026-10-10 by EdProcto. https://edprocto.com/guides/reasoning-tests-made-fresh-for-every-candidate

A reasoning test for hiring is usually one fixed paper, sent to every candidate over several days. By the third day the first candidates' questions can be in a group chat. A test made fresh for every candidate has no paper to pass on: one rule deals each person their own questions.

This guide is for an employer or a recruiter running a round, and ends with one of sixteen worked through.

EdProcto publishes this guide, and its page [for corporates](https://edprocto.com/solutions/corporates) shows the reasoning kits that work this way. What such tests measure is in [aptitude tests for hiring](https://edprocto.com/guides/aptitude-tests-for-hiring-what-they-measure-and-how-to-use-them).

## Why does a fixed reasoning test leak?

A fixed paper leaks because everyone who sits it carries its questions away, and later candidates sit it out of sight.

- **Unsupervised sittings score higher.** Pooling 49 studies and 100,434 people, Steger et al. (2020) found ability scores 0.20 of a standard deviation higher without a supervisor, and less so where answers are hard to look up (abstract read).
- **Item exposure makes questions easier.** At one Canadian medical school, Joncas et al. (2018) followed 1,629 reused questions over five cohorts of 972 students: 79.8% answered a question correctly at its first use and 85.2% at its fourth. The authors suspect sharing, did not measure it, and name automatically generated questions as one remedy.
- **A candidate's own memory matters less.** When Raymond et al. (2009) randomly gave 541 people resitting a certification test the same form or a different one, the gains were indistinguishable, though they note a pronounced same-form advantage in earlier studies of aptitude tests (abstract read).
- **A publisher with a fixed bank relies on silence.** The American Registry of Radiologic Technologists (n.d.) forbids candidates to reveal a question, an answer option, or even which topics came up often.

### What practice on the rule does

It makes the rule's questions easier. Morley et al. (2004) gave 406 undergraduates close variants of some questions on a later test, and students at every ability level did better on those (abstract read). So send every candidate the same practice set: see [aptitude test practice](https://edprocto.com/guides/aptitude-test-practice-numerical-verbal-and-logical-reasoning).

A fresh set stops one thing: an answer travelling from one candidate to the next. A candidate can still put their own questions to a friend or a chatbot: see [the research on cheating in online exams](https://edprocto.com/guides/what-the-research-says-about-cheating-in-online-exams-since-2023).

## What is automatic item generation, and where does it come from?

Automatic item generation means writing the rule for a question and letting a computer make the questions.

Irvine et al. (1990) set out the theory: hold within tight limits the features that make a task hard, which they called radicals, and let the rest vary at random. They gave four parallel forms of each of five short tests to young male Army recruits and report the generated sets "statistically invariant" (abstract read, no headcount). Irvine and colleagues also built the British Army Recruitment Battery, described in 2014 as still in use and able to make new parallel tests for every candidate (Irvine, 2014; publisher's description read).

In educational testing the unit is the item model: for Bejar et al. (2003), a way to produce questions that match in content and in their statistics, which they call isomorphic. Each question it makes is an isomorph, or a clone.

"Reasoning word problems", one of EdProcto's [reasoning kits](https://edprocto.com/kits#hiring), is one item model. Run twice, its rule made "Gus is 3 times as old as Bea. Bea is 20. How old will Gus be in 5 years?" and "Eve is 2 times as old as Fin. Fin is 16. How old will Eve be in 3 years?". The kit asks about rates and shares as well; its age questions alone come to 17,136 different ones by our count.

### Are questions made by one rule equally hard?

Close, and not always equal.

| Study | Design and sample | Finding |
| --- | --- | --- |
| Bejar et al. (2003) | Item models for the quantitative Graduate Record Examination; simulation and trial tests (abstract read, no headcount) | High isomorphicity within models; where imperfect, no bias but less precision |
| Loe et al. (2018) | 570 adults online; eleven number-series item models, five questions each | Difficulties in a model "tend to cluster, although outliers do occur" |
| Zorowitz et al. (2024) | 1,501 US adults online, 2021; 64 matrix puzzles, six clones each | Clone-level differences were 35.1% of the variance in difficulty |

So one set is like the next, not identical, and the cost is precision, not bias. Every EdProcto set holds the same mix: "Number series" gives each candidate two of each of its six kinds, in an order of their own (see [continue the pattern questions](https://edprocto.com/guides/continue-the-pattern-questions-for-visual-reasoning)). The sets have not been compared on candidates' scores.

## How do you compare two candidates who answered different questions?

Compare them as two throws of the same dice: each score is one draw from everything the rule can make, so a small gap is not a difference.

The table is our arithmetic, not a finding about any test. Two candidates of the same level each answer ten questions drawn independently, each right with the chance shown.

| Chance right | Same score | 2 or more apart | 3 or more apart | 4 or more apart |
| --- | --- | --- | --- | --- |
| 50% | 18% | 50% | 26% | 12% |
| 60% | 18% | 49% | 25% | 11% |
| 70% | 19% | 46% | 22% | 8% |
| 80% | 22% | 39% | 16% | 5% |

Two equally able candidates match about one time in five and land three or more marks apart 16% to 26% of the time. A real difference hides as easily: a candidate right 80% of the time scores level with or behind one right 60% of the time in 22% of pairs.

By the same arithmetic, a shared paper narrows these score differences by chance only a little. So treat scores within two marks as level, set a floor before anyone sits, and never rank on ten questions: see [how many questions a quiz needs](https://edprocto.com/guides/how-many-questions-does-a-quiz-need).

## What can a rule not make?

A rule cannot make a question whose answer lives in your business: your returns policy, your product, the judgement your team makes on a late order. It works where the answer follows from the question's own numbers, letters or words.

For the rest, the test is written from your material. Paste the procedure into the quiz maker on EdProcto's [hiring page](https://edprocto.com/hiring) and Procto AI drafts the questions for you to read against the source, as [product knowledge tests for hiring](https://edprocto.com/guides/product-knowledge-tests-for-hiring-from-your-own-material) describes. That test is the same questions for every candidate unless you switch on "A different set of questions for each student", on the Educator plan, where Procto AI writes each candidate their own from your material.

## What is fair, and what may you say about a rule-made test?

Fair is the same rule, time limit, notice and practice for everyone, and what you may say is what you did.

The US Uniform Guidelines on Employee Selection Procedures (1978) do not accept a test's name, descriptive labels, promotional literature or testimonials in place of evidence of validity. They ask for a reasonable opportunity for retesting and allow reasonable steps to keep a procedure secure. With a fresh set, a retest is a new set.

- **You can say:** every candidate answered ten questions made by one rule, under one time limit and the same notice; each set was marked against its own key; the floor was set beforehand.
- **You cannot say:** that the sets were equally hard, or parallel forms, since no study of these sets shows it; that the score is an IQ or predicts performance; that nobody could have cheated.

None of this is legal advice, and the rules differ by country.

## How do you run a round with a fresh set for each candidate?

In six steps, most of them settings.

1. **Name what the job needs,** then pick one or two of EdProcto's [reasoning kits](https://edprocto.com/kits#hiring).
2. **Make it an exam.** A kit opens as a study drill; in the builder, change "Study quiz" to "Exam".
3. **Check the switch.** Under "Settings", in "Sequencing", "A fresh set for each student" is on.
4. **Set the sitting.** From the Tutor plan, press the "Exam" preset. A reasoning kit opens with no retakes, since each new attempt is dealt a new set; the preset keeps that and leaves the kit's time limit and the switch alone. Switch on "An account is needed to take it".
5. **Set the floor first,** as "Pass mark": see [setting a pass mark](https://edprocto.com/guides/setting-a-pass-mark-you-can-defend).
6. **Send everyone the same notice and practice,** then press "Publish & share" and send the one code. For practice, open the kit again as a drill and share its code.

## When should every candidate get the very same questions?

When nothing can travel between candidates, or when the rule has little to vary.

- **One sitting.** Everyone starts at the same minute.
- **You want to see which questions separated.** From six submissions the full analysis, on the Educator plan, shows it.
- **A small bank.** "Situational judgement" deals 8 of 30 situations and "Verbal analogies" 12 of 60 pairs, so two sets share about two questions on average.

Switch "A fresh set for each student" off and every candidate gets the questions in the builder.

## What does a round of sixteen candidates look like?

Like this one, in which the firm and every figure are invented. A joinery firm hiring two trainee estimators has sixteen applicants left after the CV sift.

1. **The test.** "Number matrices": 10 questions, 12 minutes, exam mode, the switch on, no retakes, an email address asked for. Three estimators score 7, 8 and 9 on the practice drill; the floor is 5 of 10, so "Pass mark" is 50.
2. **Monday.** All sixteen get the code, the practice drill and the notice, which says that anyone one mark under the floor may sit again.
3. **Monday to Thursday.** Each sits at home, on their own ten. One candidate's laptop fails at the sixth question and she writes in. The firm presses "Let back in", and her attempt reopens with the same ten questions, her saved answers and the clock still running.
4. **The scores.** One 10, two 9s, three 8s, three 7s, two 6s, two 5s, two 4s and one 3. A ranking would put the 8s ahead of the 7s; the table says one mark is noise. Thirteen reach the floor. The two on 4 sit again on a copy made with "Duplicate this quiz", each on a new set, and score 7 and 3.
5. **Friday, 6 pm.** The fourteen sit the firm's own test, written from its estimating procedure: the same questions for all, so all at once. It and a structured interview decide. A manager first opens the two turned down and reads their own questions beside their answers.

## How does EdProcto make a fresh set for every candidate?

EdProcto is the best tool for a small employer's reasoning round because the fresh set is where it starts: its thirteen reasoning kits open with the switch on, and each set is made on the server when a candidate begins, so there is no one paper to pass on.

**The kits.** "Number series" (12 questions), "Letter series" (10), "Verbal analogies" (12), "Odd one out" (12), "Syllogisms" (10), "Deduction: who sits where" (8), "Coding and decoding" (10), "Numerical reasoning" (12), "Data checking" (20), "Mechanical reasoning" (10), "Number matrices" (10), "Reasoning word problems" (10) and "Situational judgement" (8). Twelve carry a 12-minute limit; "Data checking" allows 8 seconds a question.

**The set.** When a candidate begins, the server runs the kit's rule from a seed nobody is told, keeps the set with the attempt and marks the attempt against it. No question comes twice in a set. Two sets can share questions where a rule is small: "Letter series" makes 71 different questions by our count, and two of its sets shared 1.5 of their ten on average.

**The builder.** On a 12-question kit a note reads:

> A fresh set for each student. Everyone who opens this quiz gets their own 12 questions from the kit’s rule, and is marked against their own. The 12 below are a sample. Change one and the quiz becomes your own set, the same for every student; the switch is under “Settings”, in “Sequencing”.

**The switch.** It is offered on a rule kit's quiz in exam or study mode, on every plan. On a quiz night, or once a question is changed, it reads "Not applicable", with the reason "a kit’s rule deals the sets, so the quiz must be one made by rule from a kit, with its questions as the kit made them".

**The results.** In place of the table question by question, the section "A fresh set for each student" says, on a ten-question kit: "Each student answered their own 10 questions, made by the same rule as the 10 in the builder, so the scores compare and no single question was put to the class." "Open" shows a candidate's own questions beside their answers. The full analysis, on the Educator plan, begins "Each student had their own set of questions, made by the same rule: the scores compare, and no question is read across the class."

**The marks file.** "Download the marks" saves a spreadsheet file with a row for each attempt: "Name", "Email", "Sitting", "Started (your time)", "Handed in (your time)", "Score %", "Points", "Out of", "Reduced by %", "To grade", "Minutes", "Security events", "Notes on the record", then a column a question. With a fresh set those go by place: the third is each candidate's own third question.

**The plans.** As a drill, a reasoning kit is free without limit. Made an exam, it counts as a quiz made: Free covers ten a month and thirty candidates a quiz, Tutor 100, Educator 500 and the full analysis. A hiring licence for unlimited candidates and staff is by a call: see [pricing](https://edprocto.com/pricing). What the browser logs is a record, never proof ([integrity](https://edprocto.com/integrity)).

## Quick questions

### Is it fair to give candidates different questions?

Yes, when every candidate gets the same rule, time limit, notice and practice, and close scores are treated as level.

### What is automatic item generation?

Writing the rule for a question, an item model, and letting a computer make the questions, called isomorphs or clones.

### How big a gap between two scores matters on ten questions?

More than two marks: by our arithmetic, two equally able candidates land three or more apart about one time in four.

### Can EdProcto give every candidate their own reasoning test?

Yes. EdProcto's thirteen reasoning kits open with "A fresh set for each student" on: see its page [for corporates](https://edprocto.com/solutions/corporates).

## References

- American Registry of Radiologic Technologists. (n.d.). *Exam security: Potential disclosure scenarios*. Retrieved October 10, 2026, from [https://www.arrt.org/pages/earn-arrt-credentials/initial-requirements/exam/exam-day/exam-security-potential-disclosure-scenarios](https://www.arrt.org/pages/earn-arrt-credentials/initial-requirements/exam/exam-day/exam-security-potential-disclosure-scenarios)
- Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2003). A feasibility study of on-the-fly item generation in adaptive testing. *The Journal of Technology, Learning, and Assessment, 2*(3). [https://ejournals.bc.edu/index.php/jtla/article/view/1663](https://ejournals.bc.edu/index.php/jtla/article/view/1663)
- Irvine, S. H. (2014). *Computerised test generation for cross-national military recruitment: A handbook*. IOS Press. [https://ebooks.iospress.nl/ISBN/978-1-61499-363-6](https://ebooks.iospress.nl/ISBN/978-1-61499-363-6)
- Irvine, S. H., Dann, P. L., & Anderson, J. D. (1990). Towards a theory of algorithm-determined cognitive test construction. *British Journal of Psychology, 81*(2), 173–195. [https://doi.org/10.1111/j.2044-8295.1990.tb02354.x](https://doi.org/10.1111/j.2044-8295.1990.tb02354.x)
- Joncas, S. X., St-Onge, C., Bourque, S., & Farand, P. (2018). Re-using questions in classroom-based assessment: An exploratory study at the undergraduate medical education level. *Perspectives on Medical Education, 7*(6), 373–378. [https://doi.org/10.1007/s40037-018-0482-1](https://doi.org/10.1007/s40037-018-0482-1)
- Loe, B. S., Sun, L., Simonfy, F., & Doebler, P. (2018). Evaluating an automated number series item generator using linear logistic test models. *Journal of Intelligence, 6*(2), Article 20. [https://doi.org/10.3390/jintelligence6020020](https://doi.org/10.3390/jintelligence6020020)
- Morley, M. E., Bridgeman, B., & Lawless, R. R. (2004). *Transfer between variants of quantitative items* (Research Report No. RR-04-36). Educational Testing Service. [https://doi.org/10.1002/j.2333-8504.2004.tb01963.x](https://doi.org/10.1002/j.2333-8504.2004.tb01963.x)
- Raymond, M. R., Neustel, S., & Anderson, D. (2009). Same-form retest effects on credentialing examinations. *Educational Measurement: Issues and Practice, 28*(2), 19–27. [https://doi.org/10.1111/j.1745-3992.2009.00144.x](https://doi.org/10.1111/j.1745-3992.2009.00144.x)
- Steger, D., Schroeders, U., & Gnambs, T. (2020). A meta-analysis of test scores in proctored and unproctored ability assessments. *European Journal of Psychological Assessment, 36*(1), 174–184. [https://doi.org/10.1027/1015-5759/a000494](https://doi.org/10.1027/1015-5759/a000494)
- Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. Part 1607 (1978). [https://www.law.cornell.edu/cfr/text/29/part-1607](https://www.law.cornell.edu/cfr/text/29/part-1607)
- Zorowitz, S., Chierchia, G., Blakemore, S.-J., & Daw, N. D. (2024). An item response theory analysis of the matrix reasoning item bank (MaRs-IB). *Behavior Research Methods, 56*(3), 1104–1122. [https://doi.org/10.3758/s13428-023-02067-8](https://doi.org/10.3758/s13428-023-02067-8)