Building Scaffolding Dialogue Data with LLM-Simulated Novices

TL;DR

This paper introduces SimInstruct, a scalable tool that uses LLMs to simulate novice dialogues with experts, enabling the collection of high-quality scaffolding instructional data without real participants.

Contribution

The paper presents a novel expert-in-the-loop framework using LLMs to generate pedagogically rich scaffolding dialogues, enhancing data collection and model training in education.

Findings

SimInstruct dialogues are comparable to real mentoring in pedagogical relevance.

Persona traits influence expert engagement and dialogue quality.

Fine-tuned LLaMA outperforms GPT-4o in instructional quality.

Abstract

High-quality, multi-turn instructional dialogues between novices and experts are essential for developing AI systems that support teaching, learning, and decision-making. These dialogues often involve scaffolding -- the process by which an expert supports a novice's thinking through questions, feedback, and step-by-step guidance. However, such data are scarce due to privacy concerns in recording and the vulnerability inherent in help-seeking. We present SimInstruct, a scalable, expert-in-the-loop tool for collecting scaffolding dialogues. Using teaching development coaching as an example domain, SimInstruct simulates novice instructors via LLMs, varying their teaching challenges and LLM's persona traits, while human experts provide multi-turn feedback, reasoning, and instructional support. This design enables the creation of realistic, pedagogically rich dialogues without requiring real novice participants. Our results reveal that persona traits, such as extroversion and introversion, meaningfully influence how experts engage. Compared to real mentoring recordings, SimInstruct dialogues demonstrate comparable pedagogical relevance and cognitive depth. Experts also reported the process as engaging and reflective, improving both data quality and their own professional insight. We further fine-tuned a LLaMA model to be an expert model using the augmented dataset, which outperformed GPT-4o in instructional quality. Our analysis highlights GPT-4o's limitations in weak reflective questioning, overuse of generic praise, a condescending tone, and a tendency to overwhelm novices with excessive suggestions.