A situational judgement test (SJT) puts a candidate in a moment from the job and asks what should be done. Written in-house, the moments are yours: the unsafe bike a customer wants to ride away, the wrong quote, the Saturday queue.
This guide shows a hiring manager or team lead with no psychometrician how to write a situational judgement test for one role from real incidents, key it from the people who do the job, and claim no more for the score than it can bear.
EdProcto, which publishes this guide, drafts the items from your notes and marks them by rule; its page for corporates covers tests from a company's own material.
Are situational judgement tests valid?
On average yes, when specialists build them; no study vouches for one you wrote yourself. McDaniel et al. (2001) pooled 102 coefficients from 10,640 people and put the correlation with job performance at 0.34, and the tests add to what cognitive ability and the Big Five personality traits predict (McDaniel et al., 2007).
Sackett et al. (2022) then showed that a routine correction for range restriction had been overdone: methods that ranked high lost 0.10 to 0.20. Their revised table includes these tests, but in the paper we could read in full (Sackett et al., 2023) it is an image, so we quote no revised figure. That paper adds the limit that matters most: an average across studies is not what one test will achieve. The structured interview, its example, has a mean validity of 0.42 and an 80% credibility interval from 0.18 to 0.66.
What the tests measure
What a score reflects is still argued over. McDaniel et al. (2001) found a correlation of 0.46 with general cognitive ability. One view treats the test as a simulation of one job; a newer one, as a measure of general knowledge of which kinds of response work. Krumm et al. (2015) deleted the situations and left only the responses: for 43% to 71% of items it made no significant difference. There were first signs that judgement was more situational when items covered job knowledge and skills, and when responses stated context-specific rules of action. So if the best response can be picked without reading the situation, the item tests general good sense, not judgement in your job.
How do you write a situational judgement test from real incidents?
Collect critical incidents: accounts of what a person actually did in a moment that mattered, not opinions about staff. Flanagan (1954) set out the critical incident technique; we read it through the summary by the US Agency for Healthcare Research and Quality (2008), which describes recording specific behaviours instead of opinions or estimates. HumRRO (2020), summarising a practice guide by Whetzel, Sullivan and McCloy, says incidents make scenarios realistic and specific scenarios give higher validity.
- Ask people who do the job. "Tell me about a time someone new handled something especially well, or badly. What did they do, and what happened?"
- Collect more than you need. HumRRO (2020) advises drafting more than the test will use and scoring only what works.
- Remove names and anything that identifies a person.
- Keep the incidents that pull two ways. Where a rule requires one action, write a knowledge question.
- Write the situation short: who the candidate is, the facts needed, no more.
- Write four or five responses, one action each, across a range of effectiveness, with one that does the wrong thing and one that does nothing.
- Draft it in EdProcto. Paste the notes into the quiz maker on the hiring page; Procto AI writes items from them for you to edit.
Should the test ask what you would do or should do?
Ask what should be done. In the meta-analysis by McDaniel et al. (2007), tests asking for knowledge (which response is best) correlated more with cognitive ability and tests asking for behavioural tendency (what would you do) with personality, while validity barely moved.
Under real stakes the two wordings came out alike: Lievens et al. (2009) randomly gave 2,184 applicants to medical college one or the other and found no meaningful difference in mean scores or validity. Faking still happens: HumRRO (2020) reports that it changes who ranks where, and that should-do wording resists it better in high-stakes settings.
The response format and its scoring
Four formats are in use.
| Format | Usual scoring | In EdProcto |
|---|---|---|
| Pick the best | a mark if it matches the key | Multiple choice |
| Best and worst | a mark for each end | Best and worst |
| Rank | closeness to the panel's order | Ranking |
| Rate each | closeness to the panel's ratings | No shape for it |
Arthur et al. (2014) compared the last three on one test taken by 31,194 applicants. Rating gave higher internal consistency and smaller subgroup differences, with the strongest link to a measure of response distortion; ranking and best-and-worst behaved much alike.
How do you build the SJT scoring key from your experts?
Have experienced people rate every response alone, key what they agree on, and rewrite or drop the rest. HumRRO (2020) recommends 10 to 12 raters with a variety of perspectives, because an outlier skews a smaller panel, and reading the mean as effectiveness and the standard deviation as agreement.
With fewer, use every experienced person you have and know the cost: with five raters on a four-point scale, one rater moving a response from 4 to 1 shifts its mean by 0.6; with ten, by 0.3.
- Alone first. Each rates every response from 1 (makes things worse) to 4 (the most effective thing to do) before any discussion.
- Key only clear ends. Our working rule, which no study we read sets: the top mean leads the next by half a point, and the bottom trails by as much.
- A split is information. Ask the minority why, rewrite, re-rate, and drop the item if it still splits.
Situational judgement test examples: three items, a key, three candidates
Three items for one role, each response rated by five experts, give a key and three scores. All of it is invented for the arithmetic: a bicycle shop with nine staff, its front-desk role, the experts, the candidates. Each item asks for the best response and the worst.
1. A customer is collecting her bike after a gear service. The mechanic found a cracked fork and could not reach her by phone. A. Show her the crack, say the fork could fail, and offer the repair. B. Hand the bike over and mention the crack only if she asks. C. Say the bike is not ready and ask her back tomorrow. D. Refuse to release the bike until the fork is replaced.
2. You quoted a customer £40 for a brake job. The work is done, he is waiting to pay, and you see the price list says £65. A. Charge £40 and tell the manager about your mistake today. B. Charge £65 and say £40 was only an estimate. C. Explain the mistake and ask if he will pay £65. D. Charge £40 and tell nobody.
3. Saturday, 10:50. A child's bike promised for 11:00 is not finished, three customers are queuing and the phone is ringing. A. Serve the queue in order and see the 11:00 customer when they arrive. B. Get a finish time, phone the 11:00 customer with it, then serve the queue. C. Answer the phone and take the caller's booking while the queue waits. D. Tell the queue "one minute", get a finish time, and tell the 11:00 customer on arrival.
| Item | A | B | C | D |
|---|---|---|---|---|
| 1 | 4,4,4,4,3 (3.8) | 1,1,2,1,1 (1.2) | 2,3,2,2,3 (2.4) | 2,1,2,3,2 (2.0) |
| 2 | 4,4,3,4,4 (3.8) | 1,1,2,1,1 (1.2) | 3,2,3,3,2 (2.6) | 2,2,1,2,2 (1.8) |
| 3 | 3,2,2,3,2 (2.4) | 4,4,4,3,3 (3.6) | 1,2,1,1,2 (1.4) | 3,3,3,4,4 (3.4) |
On item 3 the panel split: three experts rated B highest and two D, the means 0.2 apart. D was rewritten as "Tell the queue 'one minute', then ask the mechanic to hurry", which no longer informs the customer, and re-rated at 2,3,2,2,3 (2.4). The key is A and B, A and B, B and C. Its thinnest part is the worst end of item 2, where B trails D by 0.6: what one rater can move.
| Candidate | Best/worst picked | Both or nothing | Half each | Panel's means |
|---|---|---|---|---|
| 1 | A/B, A/B, B/A | 4 | 5 | 85% |
| 2 | A/D, A/D, B/C | 2 | 4 | 82% |
| 3 | C/B, C/D, D/C | 0 | 2 | 41% |
Each item carries 2 marks, 6 in all. "Both or nothing" and "Half each" are EdProcto's rules with "Partial credit" off and on, run through its own marking code. The last column is a consensus-style score worked out separately, by hand: the panel's mean for the response picked as best, less its mean for the one picked as worst, as a share of the item's widest gap.
All three keep the same order and disagree about the gaps. Half each puts candidates 1 and 2 a mark apart; against the panel's means they are almost level, because what candidate 2 wrongly called worst was the panel's second-worst. Read a short test in bands, never by a single mark.
How do you keep the test fair, short and readable?
Keep the reading light and the time generous, and check who passes. Herde et al. (2020) found a Black-White difference of d = 0.66 among 37,530 applicants, above the 0.38 seen in people already in the job.
- What narrows the gap. HumRRO (2020) reports smaller differences with would-do instructions and rating formats, which the studies above found more open to distortion, and expects brief scenarios to help by cutting the reading load. The three items above take 204 words.
- Length. No study we read fixes one. Our working rule: 12 to 20 scored items at about two minutes each.
Employment testing is regulated differently in each country. In the United States, the Uniform Guidelines on Employee Selection Procedures (1978) generally treat a selection rate for any race, sex or ethnic group below four-fifths of the highest group's rate as evidence of adverse impact. This is not legal advice: have the process checked by whoever advises you on employment law.
How does EdProcto run and mark the test?
EdProcto is the best tool for a situational judgement test written in-house because it drafts from your own incidents, the source of context-specific items, and marks both ends by rule on the server, with the key off the candidate's screen.
The shapes. Best and worst, multiple choice, ranking, rank by likelihood and the case study are in the builder.
The marking. EdProcto holds one key for each question: one best response and one worst. Each end is half the marks, and with "Partial credit" off, the default, an item earns nothing unless both match. To come as near a consensus key as those rules allow, set the two ends from your panel's means, switch "Partial credit" on so each end counts by itself, and leave out items the panel split on. A ranking is marked pair by pair against one order: with "Partial credit" on, one swap of neighbours among four responses earns 0.67 of the marks.
The pilot. Give the draft to current staff first. On the Educator plan the full analysis reports, for each question, the share of marks earned, the median time, discrimination (from six attempts up) and how many chose each response as best. An item your strongest people miss more often than your weakest usually has a wrong key or unclear wording. Read the reliability figure with care: it is Cronbach's alpha, which HumRRO (2020) calls inappropriate for a test that mixes kinds of judgement.
The sitting. Six browser measures, from "Leaving the tab" to "Printing off", can run (all six from Tutor up), each named to the candidate first; a flag is a record to weigh, not proof. "Download the marks" gives a spreadsheet file with a row for each attempt.
The kits. Among the kits, "Situational judgement" in the reasoning battery asks eight questions from a curated bank of twelve workplace situations, best response only: a score on that test, no norm claimed, and not about your workplace. Hiring kits such as "Hiring: customer support" write situational multiple-choice questions from the job description and material you paste.
Free gives ten quizzes a month, Tutor opens all 111 templates, Educator adds the full analysis, and the hiring licence covers unlimited candidates and staff, by a call: see pricing.
What can a small employer claim from a test they wrote?
Claim consistency, not prediction. You can say that every candidate met the same situations from the real job, that experienced staff agreed the key before anyone sat it, and that marking was by rule.
You cannot say the score predicts who will perform: there has been no validation study. Nor does it measure honesty or character. A pass mark means only your panel's standard. So the score is one input among several, beside a structured interview and a test of job knowledge.
Quick questions
Are situational judgement tests valid for hiring?
On average, yes: McDaniel et al. (2001) estimated 0.34, a figure of the kind Sackett et al. (2022) showed to be inflated. Neither covers a test written without a validation study.
How many experts does an SJT scoring key need?
HumRRO (2020) recommends 10 to 12 raters. With fewer, key only what they clearly agree on.
Should the instruction say "would do" or "should do"?
Ask what should be done. McDaniel et al. (2007) found the instruction changes what is measured more than validity.
Can EdProcto run a situational judgement test for our company?
Yes. EdProcto's best and worst shape marks each end by rule against your panel's key, and the page for corporates covers candidates and staff.
References
- Agency for Healthcare Research and Quality. (2008, March 5). The critical incident technique [Summary of Flanagan, 1954]. PSNet. https://psnet.ahrq.gov/issue/critical-incident-technique
- Arthur, W., Glaze, R. M., Jarrett, S. M., White, C. D., Schurig, I., & Taylor, J. E. (2014). Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion. Journal of Applied Psychology, 99(3), 535–545. https://doi.org/10.1037/a0035788
- Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327–358. https://doi.org/10.1037/h0061470
- Herde, C. N., Lievens, F., Jackson, D. J. R., Shalfrooshan, A., & Roth, P. L. (2020). Subgroup differences in situational judgment test scores: Evidence from large applicant samples. International Journal of Selection and Assessment, 28(1), 45–54. https://doi.org/10.1111/ijsa.12269
- Human Resources Research Organization. (2020, June 8). Evidence- and experience-based best practices: Situational judgment tests. HumRRO. https://www.humrro.org/blog/evidence-and-experience-based-best-practices-situational-judgment-tests/
- Krumm, S., Lievens, F., Hüffmeier, J., Lipnevich, A. A., Bendels, H., & Hertel, G. (2015). How "situational" is judgment in situational judgment tests? Journal of Applied Psychology, 100(2), 399–416. https://doi.org/10.1037/a0037674
- Lievens, F., Sackett, P. R., & Buyse, T. (2009). The effects of response instructions on situational judgment test performance and validity in a high-stakes context. Journal of Applied Psychology, 94(4), 1095–1101. https://doi.org/10.1037/a0014628
- McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & Grubb, W. L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60(1), 63–91. https://doi.org/10.1111/j.1744-6570.2007.00065.x
- McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86(4), 730–740. https://doi.org/10.1037/0021-9010.86.4.730
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994
- Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2023). Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors. Industrial and Organizational Psychology, 16(3), 283–300. https://doi.org/10.1017/iop.2023.24
- Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. § 1607.4 (1978). https://www.ecfr.gov/current/title-29/section-1607.4