Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors

TL;DR

This paper introduces TRAVER, a novel turn-by-turn verification approach for coding tutoring agents powered by LLMs, enhancing their ability to guide students effectively in complex tasks.

Contribution

It proposes the Trace-and-Verify (TRAVER) framework combining knowledge tracing and turn-by-turn verification, along with DICT for automatic evaluation of tutoring agents.

Findings

TRAVER significantly improves success rates in coding tutoring tasks.

The DICT evaluation protocol effectively assesses tutoring agent performance.

Experiments highlight challenges and potential of LLMs in complex educational tasks.

Abstract

Intelligent tutoring agents powered by large language models (LLMs) have been increasingly explored to deliver personalized knowledge in areas such as language learning and science education. However, their capabilities in guiding users to solve complex real-world tasks remain underexplored. To address this limitation, in this work, we focus on coding tutoring, a challenging problem that requires tutors to proactively guide students towards completing predefined coding tasks. We propose a novel agent workflow, Trace-and-Verify (TRAVER), which combines knowledge tracing to estimate a student's knowledge state and turn-by-turn verification to ensure effective guidance toward task completion. We introduce DICT, an automatic evaluation protocol that assesses tutor agents using controlled student simulation and code generation tests. Extensive experiments reveal the challenges of coding tutoring and demonstrate that TRAVER achieves a significantly higher success rate. Although we use code tutoring as an example in this paper, our approach can be extended beyond coding, providing valuable insights into advancing tutoring agents for human task learning.