Assessment Design in the AI Era: A Method for Identifying Items Functioning Differentially for Humans and Chatbots

TL;DR

This paper introduces a statistically grounded method using Differential Item Functioning analysis to identify assessment items where humans and chatbots differ, aiding in designing fair and valid evaluations in the era of AI.

Contribution

It combines educational data mining and psychometric theory to systematically detect items vulnerable to AI misuse and characterize task features influencing chatbot performance.

Findings

DIF analysis effectively identifies items with systematic human-chatbot response differences.

The method reveals task dimensions that make problems easier or harder for AI.

DIF-informed analytics enhance understanding of AI-human capability divergence.

Abstract

The rapid adoption of large language models (LLMs) in education raises profound challenges for assessment design. To adapt assessments to the presence of LLM-based tools, it is crucial to characterize the strengths and weaknesses of LLMs in a generalizable, valid and reliable manner. However, current LLM evaluations often rely on descriptive statistics derived from benchmarks, and little research applies theory-grounded measurement methods to characterize LLM capabilities relative to human learners in ways that directly support assessment design. Here, by combining educational data mining and psychometric theory, we introduce a statistically principled approach for identifying items on which humans and LLMs show systematic response differences, pinpointing where assessments may be most vulnerable to AI misuse, and which task dimensions make problems particularly easy or difficult for generative AI. The method is based on Differential Item Functioning (DIF) analysis -- traditionally used to detect bias across demographic groups -- together with negative control analysis and item-total correlation discrimination analysis. It is evaluated on responses from human learners and six leading chatbots (ChatGPT-4o \& 5.2, Gemini 1.5 \& 3 Pro, Claude 3.5 \& 4.5 Sonnet) to two instruments: a high school chemistry diagnostic test and a university entrance exam. Subject-matter experts then analyzed DIF-flagged items to characterize task dimensions associated with chatbot over- or under-performance. Results show that DIF-informed analytics provide a robust framework for understanding where LLM and human capabilities diverge, and highlight their value for improving the design of valid, reliable, and fair assessment in the AI era.