Beyond Static Scoring: Enhancing Assessment Validity via AI-Generated Interactive Verification

TL;DR

This paper proposes a Human-AI collaboration framework that combines automated scoring with AI-generated follow-up questions to improve assessment validity and authenticity beyond static scoring methods.

Contribution

It introduces a novel interactive verification approach that integrates AI-generated questions with automated scoring to enhance assessment validity and detect superficial reasoning.

Findings

Stage 1 ensures procedural fairness and consistency.

Stage 2 effectively diagnoses superficial reasoning.

Instructor perceptions highlight the importance of adaptive questioning.

Abstract

Large Language Models (LLMs) challenge the validity of traditional open-ended assessments by blurring the lines of authorship. While recent research has focused on the accuracy of automated scoring (AES), these static approaches fail to capture process evidence or verify genuine student understanding. This paper introduces a novel Human-AI Collaboration framework that enhances assessment integrity by combining rubric-based automated scoring with AI-generated, targeted follow-up questions. In a pilot study with university instructors (N=9), we demonstrate that while Stage 1 (Auto-Scoring) ensures procedural fairness and consistency, Stage 2 (Interactive Verification) is essential for construct validity, effectively diagnosing superficial reasoning or unverified AI use. We report on the systems design, instructor perceptions of fairness versus validity, and the necessity of adaptive difficulty in follow-up questioning. The findings offer a scalable pathway for authentic assessment that moves beyond policing AI to integrating it as a synergistic partner in the evaluation process.