Leveraging Large Language Models to Identify Conversation Threads in Collaborative Learning

TL;DR

This paper explores how explicit conversation threading improves large language models' ability to analyze synchronous group dialogues, enhancing understanding of collaborative learning interactions.

Contribution

It provides a systematic guide for identifying conversation threads and benchmarks LLM prompting strategies for better discourse analysis in real-time group talks.

Findings

Thread information improves LLM coding accuracy

Well-structured dialogue enhances downstream analysis

Hybrid human-AI approaches offer practical benefits

Abstract

Understanding how ideas develop and flow in small-group conversations is critical for analyzing collaborative learning. A key structural feature of these interactions is threading, the way discourse talk naturally organizes into interwoven topical strands that evolve over time. While threading has been widely studied in asynchronous text settings, detecting threads in synchronous spoken dialogue remains challenging due to overlapping turns and implicit cues. At the same time, large language models (LLMs) show promise for automating discourse analysis but often struggle with long-context tasks that depend on tracing these conversational links. In this paper, we investigate whether explicit thread linkages can improve LLM-based coding of relational moves in group talk. We contribute a systematic guidebook for identifying threads in synchronous multi-party transcripts and benchmark different LLM prompting strategies for automated threading. We then test how threading influences performance on downstream coding of conversational analysis frameworks, that capture core collaborative actions such as agreeing, building, and eliciting. Our results show that providing clear conversational thread information improves LLM coding performance and underscores the heavy reliance of downstream analysis on well-structured dialogue. We also discuss practical trade-offs in time and cost, emphasizing where human-AI hybrid approaches can yield the best value. Together, this work advances methods for combining LLMs and robust conversational thread structures to make sense of complex, real-time group interactions.