Developing a Situational Judgement Test to Assess Clinical Judgement in Fourth-Year Medical Students: A Pilot Study

TL;DR

This study created and tested a situational judgment test to evaluate clinical judgment in fourth-year medical students, finding it to be a promising but not yet fully validated assessment tool.

Contribution

The paper introduces a new situational judgment test for assessing clinical judgment in medical students and evaluates its psychometric properties.

Findings

The SJT showed acceptable internal consistency when using the distance-from-SME-best-answer-squared scoring method.

Most students preferred the SJT over traditional multiple-choice exams and felt it was appropriate for assessing clinical judgment.

Item analysis revealed mixed results, with some scenarios having good discrimination and difficulty indices.

Abstract

Introduction

Assessing clinical judgement objectively and economically presents a challenge in academic medicine. The authors developed a situational judgement test (SJT) to measure fourth-year medical students’ clinical judgement.

Methods

A knowledge-based, single-best-answer SJT was developed by a panel of subject matter experts (SMEs). The SJT included 30 scenarios, each with five response options ranked ordinally from most to least appropriate. A computer-based format was used, and the SJT was piloted by two cohorts of fourth-year medical students at California University of Science and Medicine in 2022 and 2023 upon completion of an internship preparation course. Subsequently, students completed an optional survey. Evaluated scoring methods included original ordinal ranking, dichotomous, dichotomous with negative correction, distance from SME best answer, and distance from SME best answer squared.

Results

The SJT was completed by 142 fourth-year medical students. Cronbach’s alpha ranged from 0.39 to 0.85, depending on the scoring method used. The distance-from-SME-best-answer-squared method yielded the highest internal consistency, which was considered acceptable. Using this scoring method, the mean score was 72.89 (SD = 48.32, range = 26-417), and the standard error of measurement was 18.41. Item analysis found that seven (23%) scenarios were of average difficulty, 13 (43%) had a good or satisfactory discrimination index, and nine (30%) had a distractor efficiency of at least 66%. Most students preferred the SJT to a traditional multiple-choice exam (16; 62%) and felt it was an appropriate tool to assess clinical judgement (15; 58%).

Conclusions

The authors developed and piloted an SJT to assess clinical judgement among medical students. Although not achieving validation, subsequent development of the SJT will focus on expanding the SME concordance panel, improving difficulty and discrimination indices, and conducting parallel forms reliability and adverse impact analyses.

Figures