The Test Your Organization Might Not Be Using, but Should
Have you heard feedback that your training participants “don’t like” the tests used in your training programs? In the training industry, many foundational courses continue to use multiple-choice tests quite effectively to ensure participants understand course material. However, for hands-on courses where participants learn a skill, the evaluation type should match the content of what participants learn. For example, the best evaluation of a skill such as data management would require participants to complete a data management exercise. For a course teaching conflict management, participants would take part in a role play requiring them to mediate between two employees.
Simulations and exercises such as these are often expensive and time-intensive to develop, administer, and score, so training programs mainly do without them. Knowing this, how should those of us in the learning and development industry proceed when we have competing interests of a high-quality, reliable, and valid testing approach that doesn’t involve excessive development time and budget?
One effective approach is to use a situational judgment test (SJT). SJTs present test-takers with written or video-based scenarios, and ask them to respond in one of two ways: (1) choose how they would likely behave in a given situation, or (2) evaluate the effectiveness of possible responses to a given situation. In the first response type, test-takers can choose the best or worst response from a list of potential responses, rate how effective each potential response option is, or even rank the potential responses in order of effectiveness.
To develop the “situations,” or scenarios, and the list of potential response options, test developers work with subject matter experts (SMEs) in the area they intend to test. For example, in an SJT of leadership skills, test developers would work with SMEs to gather information about situations where performance is particularly good, or particularly bad, and then develop a list of potential responses to the scenario. These types of scenarios are called “critical incidents”. It can even be relatively easy to write these scenarios, as those of us without explicit SME-level expertise in leadership have seen plenty of situations, good and bad.
Because SJTs are a method of assessment, and not an assessment of a specific construct (e.g., personality, intelligence, or work ethic), SJTs are flexible enough to test many characteristics, such as:
- Critical thinking and reasoning
- Supervisory potential
- Judgment in work-related situations
- Interpersonal skills
SJTs can be video-based or paper-based; video-based SJTs have been promising for assessing interpersonal skills, although the additional expense may make video-based SJTs prohibitive for most learning and development organizations.
SJTs have immense value to organizations—they predict on-the-job performance, and in some cases, they predict performance 7-9 years later. They demonstrate less adverse impact against minority test-takers than typical tests of cognitive ability, and are less susceptible to faking than other types of assessments like personality instruments. SJT items that assess personality ask test takers to indicate the response option that most often reflects what they would do, as opposed to asking participants to self-report on their own characteristics, making it more difficult to fake. Finally, SJTs are more cost effective than high-fidelity simulations like role-play exercises, but have similar validity for predicting job performance.
An SJT is a good training evaluation option because responses simulate judgment processes in the work context, and can serve as proxy measures of transfer of training when participants rate their response options according to their likelihood to respond to the situation (Hauenstein, Findlay, & McDonald, 2010). Finally, SJT items can help learning and development groups diagnose problems with the aspects of training, such as the content, instruction, or stated difficulty of the training program, by tracking training participants’ improvement on SJT items after the course.
To help your training organization develop tests that your training participants have more favorable reactions to, while also providing valid and reliable evaluations, consider SJTs instead of multiple-choice tests or costly simulations. SJTs are often a more appropriate test than a multiple choice test for such competencies as:
SJTs are a valid, reliable, and acceptable albeit underused test type for evaluating learning in training programs, and they don’t have to break the development budget.