From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding

TL;DR

This paper introduces an annotated dataset of student dialogues with discourse features and establishes baseline models using GPT-3.5 and Llama-3.1 to automatically detect discourse, highlighting challenges and future research potential.

Contribution

It provides a new educational dialogue dataset with discourse annotations and baseline models for automatic discourse detection in student conversations.

Findings

Baseline models perform suboptimally on discourse prediction.

Annotated dataset enables future NLP research in educational dialogue analysis.

Highlights the gap between NLP models and educational discourse understanding.

Abstract

Identifying discourse features in student conversations is quite important for educational researchers to recognize the curricular and pedagogical variables that cause students to engage in constructing knowledge rather than merely completing tasks. The manual analysis of student conversations to identify these discourse features is time-consuming and labor-intensive, which limits the scale and scope of studies. Leveraging natural language processing (NLP) techniques can facilitate the automatic detection of these discourse features, offering educational researchers scalable and data-driven insights. However, existing studies in NLP that focus on discourse in dialogue rarely address educational data. In this work, we address this gap by introducing an annotated educational dialogue dataset of student conversations featuring knowledge construction and task production discourse. We also establish baseline models for automatically predicting these discourse properties for each turn of talk within conversations, using pre-trained large language models GPT-3.5 and Llama-3.1. Experimental results indicate that these state-of-the-art models perform suboptimally on this task, indicating the potential for future research.