Towards Mining Effective Pedagogical Strategies from Learner-LLM Educational Dialogues

TL;DR

This paper introduces a dialogue analysis approach to identify effective pedagogical strategies from learner-LLM interactions, emphasizing the importance of dialogue dynamics in evaluating educational LLM applications.

Contribution

It proposes a novel methodology combining dialogue act annotation, pattern mining, and predictive modeling to analyze learner-LLM dialogues for pedagogical insights.

Findings

Initial dialogue data collection completed

Preliminary dialogue act patterns identified

Early insights suggest potential pedagogical strategies

Abstract

Dialogue plays a crucial role in educational settings, yet existing evaluation methods for educational applications of large language models (LLMs) primarily focus on technical performance or learning outcomes, often neglecting attention to learner-LLM interactions. To narrow this gap, this AIED Doctoral Consortium paper presents an ongoing study employing a dialogue analysis approach to identify effective pedagogical strategies from learner-LLM dialogues. The proposed approach involves dialogue data collection, dialogue act (DA) annotation, DA pattern mining, and predictive model building. Early insights are outlined as an initial step toward future research. The work underscores the need to evaluate LLM-based educational applications by focusing on dialogue dynamics and pedagogical strategies.