Reflecting in the Reflection: Integrating a Socratic Questioning Framework into Automated AI-Based Question Generation

TL;DR

This paper presents a novel automated framework for generating high-quality reflection questions by simulating a Socratic dialogue between specialized AI agents, improving question relevance and depth in educational settings.

Contribution

It introduces a two-agent Socratic questioning framework that iteratively refines questions, demonstrating superior quality over one-shot methods in educational question generation.

Findings

Dynamic stopping improves question quality over fixed iterations.

Two-agent protocol yields more relevant and deeper questions.

Long dialogues can lead to drift or over-complication.

Abstract

Designing good reflection questions is pedagogically important but time-consuming and unevenly supported across teachers. This paper introduces a reflection-in-reflection framework for automated generation of reflection questions with large language models (LLMs). Our approach coordinates two role-specialized agents, a Student-Teacher and a Teacher-Educator, that engage in a Socratic multi-turn dialogue to iteratively refine a single question given a teacher-specified topic, key concepts, student level, and optional instructional materials. The Student-Teacher proposes candidate questions with brief rationales, while the Teacher-Educator evaluates them along clarity, depth, relevance, engagement, and conceptual interconnections, responding only with targeted coaching questions or a fixed signal to stop the dialogue. We evaluate the framework in an authentic lower-secondary ICT setting on the topic, using GPT-4o-mini as the backbone model and a stronger GPT- 4-class LLM as an external evaluator in pairwise comparisons of clarity, relevance, depth, and overall quality. First, we study how interaction design and context (dynamic vs.fixed iteration counts; presence or absence of student level and materials) affect question quality. Dynamic stopping combined with contextual information consistently outperforms fixed 5- or 10-step refinement, with very long dialogues prone to drift or over-complication. Second, we show that our two-agent protocol produces questions that are judged substantially more relevant and deeper, and better overall, than a one-shot baseline using the same backbone model.