Using LLMs to identify features of personal and professional skills in an open-response situational judgment test

TL;DR

This paper explores using large language models to automatically extract relevant features from open-response situational judgment tests, aiming to improve scalable assessment of personal and professional skills.

Contribution

It introduces a novel method leveraging LLMs to extract construct-relevant features from SJT responses, addressing previous limitations of NLP scoring systems.

Findings

LLMs can effectively identify relevant features in SJT responses

The approach shows promise for scalable, automated scoring of personal skills

Foundation laid for future automated assessment tools

Abstract

Academic programs are increasingly recognizing the importance of personal and professional skills and their critical role alongside technical expertise in preparing students for future success in diverse career paths. With this growing demand comes the need for scalable systems to measure, evaluate, and develop these skills. Situational Judgment Tests (SJTs) offer one potential avenue for measuring these skills in a standardized and reliable way, but open-response SJTs have traditionally relied on trained human raters for evaluation, presenting operational challenges to delivering SJTs at scale. Past attempts at developing NLP-based scoring systems for SJTs have fallen short due to issues with construct validity of these systems. In this article, we explore a novel approach to extracting construct-relevant features from SJT responses using large language models (LLMs). We use the Casper SJT to demonstrate the efficacy of this approach. This study sets the foundation for future developments in automated scoring for personal and professional skills.