gencat: Generative computerized adaptive testing

TL;DR

GENCAT introduces a novel framework using large language models and generative item response theory to improve adaptive testing, especially for open-ended questions, outperforming existing methods in real-world programming datasets.

Contribution

The paper presents GENCAT, a new generative CAT framework leveraging LLMs and GIRT for better knowledge estimation and question selection, especially for open-ended responses.

Findings

Outperforms existing CAT baselines in real-world datasets.

Achieves up to 4.32% AUC improvement in early testing stages.

Effectively estimates student knowledge from open-ended responses.

Abstract

Existing computerized Adaptive Testing (CAT) frameworks are typically built on predicting the correctness of a student response to a question. Although effective, this approach fails to leverage textual information in questions and responses, especially for open-ended questions. In this work, we propose GENCAT (\textbf{GEN}erative \textbf{CAT}), a novel CAT framework that leverages Large Language Models for knowledge estimate and question selection. First, we develop a Generative Item Response Theory (GIRT) model that enables us to estimate student knowledge from their open-ended responses and predict responses to unseen questions. We train the model in a two-step process, first via Supervised Fine-Tuning and then via preference optimization for knowledge-response alignment. Second, we introduce three question selection algorithms that leverage the generative capabilities of the GIRT model, based on the uncertainty, linguistic diversity, and information of sampled student responses. Third, we conduct experiments on two real-world programming datasets and demonstrate that GENCAT outperforms existing CAT baselines, achieving an AUC improvement of up to 4.32\% in the key early testing stages.