Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systems

TL;DR

This paper presents Ruffle&Riley, an innovative conversational tutoring system that automatically generates and orchestrates tutoring scripts using large language models, aiming to reduce development costs and enhance learning experiences.

Contribution

It introduces a novel LLM-based framework for automatic induction and orchestration of conversational tutoring systems, advancing scalable and adaptable educational technology.

Findings

No significant difference in post-test scores compared to simpler chatbots.

Users reported higher understanding and memory retention.

Perceived support and coherence were rated higher by Ruffle&Riley users.

Abstract

Conversational tutoring systems (CTSs) offer learning experiences driven by natural language interaction. They are known to promote high levels of cognitive engagement and benefit learning outcomes, particularly in reasoning tasks. Nonetheless, the time and cost required to author CTS content is a major obstacle to widespread adoption. In this paper, we introduce a novel type of CTS that leverages the recent advances in large language models (LLMs) in two ways: First, the system induces a tutoring script automatically from a lesson text. Second, the system automates the script orchestration via two LLM-based agents (Ruffle&Riley) with the roles of a student and a professor in a learning-by-teaching format. The system allows a free-form conversation that follows the ITS-typical inner and outer loop structure. In an initial between-subject online user study (N = 100) comparing Ruffle&Riley to simpler QA chatbots and reading activity, we found no significant differences in post-test scores. Nonetheless, in the learning experience survey, Ruffle&Riley users expressed higher ratings of understanding and remembering and further perceived the offered support as more helpful and the conversation as coherent. Our study provides insights for a new generation of scalable CTS technologies.