Dude, where's my utterance? Evaluating the effects of automatic segmentation and transcription on CPS detection

TL;DR

This paper evaluates how automatic transcription and segmentation affect the detection of collaborative problem-solving markers, finding comparable performance to manual methods but with reduced utterance granularity, impacting AI tools for education.

Contribution

It systematically assesses the impact of automated speech processing components on CPS detection accuracy in educational settings.

Findings

Automated transcription and segmentation yield CPS detection comparable to manual methods.

Automated segmentation reduces utterance count by 26.5%, affecting data granularity.

Implications discussed for AI tools supporting collaborative learning.

Abstract

Collaborative Problem-Solving (CPS) markers capture key aspects of effective teamwork, such as staying on task, avoiding interruptions, and generating constructive ideas. An AI system that reliably detects these markers could help teachers identify when a group is struggling or demonstrating productive collaboration. Such a system requires an automated pipeline composed of multiple components. In this work, we evaluate how CPS detection is impacted by automating two critical components: transcription and speech segmentation. On the public Weights Task Dataset (WTD), we find CPS detection performance with automated transcription and segmentation methods is comparable to human-segmented and manually transcribed data; however, we find the automated segmentation methods reduces the number of utterances by 26.5%, impacting the the granularity of the data. We discuss the implications for developing AI-driven tools that support collaborative learning in classrooms.