Does the TalkMoves Codebook Generalize to One-on-One Tutoring and Multimodal Interaction?

TL;DR

This study evaluates whether the TalkMoves codebook, designed for classroom discourse, effectively generalizes to one-on-one tutoring across multimodal data, revealing limitations and the need for modality-aware tools.

Contribution

It compares the traditional TalkMoves codebook with a hybrid AI-human approach in tutoring settings, highlighting their reliability, coverage, and usability differences.

Findings

TalkMoves achieved higher inter-rater reliability (k=0.74) than the AI-human codebook (k=0.64).

The AI-human codebook showed broader empirical coverage and higher usability.

Both codebooks undercaptured tutoring-relevant moves and faced ambiguity with nonverbal cues.

Abstract

Accountable Talk theory has been widely adopted to analyze classroom discourse and is increasingly used to annotate tutoring interactions. In particular, the TalkMoves codebook, grounded in Accountable Talk theory, is commonly used to label tutoring data and train models of effective instructional support. However, Accountable Talk was originally developed to characterize collaborative, whole-classroom oral discourse, not to identify talk moves in one-on-one tutoring environments using multimodal data (e.g., video, audio, chat). As tutoring platforms expand in scale and modality, questions remain about whether Accountable Talk-based codebooks generalize reliably beyond their original classroom context and data representation. This study examines whether the human-developed TalkMoves codebook generalizes in reliability, utility, and interpretability when applied to one-on-one tutoring across audio, chat, and multimodal data. We compare TalkMoves with a hybrid AI-human developed codebook using a workflow established in prior research. Two expert annotators with over 20 years of teaching experience applied both codebooks to six tutoring sessions spanning three modalities: chat-based, audio-only, and multimodal interactions. Results show that while Talk-Moves achieved higher overall inter-rater reliability than the AI-human codebook (k = 0.74 vs. 0.64), the AI-human codebook demonstrated broader empirical coverage and higher perceived usability across modalities. Both codebooks undercaptured tutoring-relevant moves and introduced ambiguity when identifying actions expressed through nonverbal and multimodal artifacts. Together, these findings highlight the uneven generalizability of TalkMoves to tutoring contexts and motivate the development of modality-aware, tutoring-grounded codebooks.