The Conversational Exam: A Scalable Assessment Design for the AI Era

TL;DR

The paper introduces the conversational exam, a scalable oral assessment method that enhances validity by requiring students to code and explain their reasoning live, addressing challenges posed by AI-generated work.

Contribution

It presents a practical, scalable oral exam format that combines authentic coding practice with real-time assessment, supported by implementation guidance.

Findings

Oral exams can be scaled to typical class sizes.

The format ensures assessment validity through live coding and explanation.

Students demonstrated genuine engagement during the exam.

Abstract

Traditional assessment methods collapse when students use generative AI to complete work without genuine engagement, creating an illusion of competence where they believe they're learning but aren't. This paper presents the conversational exam -- a scalable oral examination format that restores assessment validity by having students code live while explaining their reasoning. Drawing on human-computer interaction principles, we examined 58 students in small groups across just two days, demonstrating that oral exams can scale to typical class sizes. The format combines authentic practice (students work with documentation and supervised AI access) with inherent validity (real-time performance cannot be faked). We provide detailed implementation guidance to help instructors adapt this approach, offering a practical path forward when many educators feel paralyzed between banning AI entirely or accepting that valid assessment is impossible.