Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

TL;DR

This paper introduces a computational framework to measure teacher uptake of student contributions in classrooms, using a new dataset and a divergence-based metric that correlates with educational outcomes.

Contribution

It presents a novel dataset, formalizes uptake as pJSD, and demonstrates its effectiveness and generalizability across multiple educational datasets.

Findings

pJSD outperforms repetition-based baselines in identifying uptake.

pJSD correlates significantly with instruction quality across datasets.

Repetition captures part of uptake but is less comprehensive.

Abstract

In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers' uptake of student contributions has been linked to higher student achievement. Yet measuring and improving teachers' uptake at scale is challenging, as existing methods require expensive annotation by experts. We propose a framework for computationally measuring uptake, by (1) releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts; (2) formalizing uptake as pointwise Jensen-Shannon Divergence (pJSD), estimated via next utterance classification; (3) conducting a linguistically-motivated comparison of different unsupervised measures and (4) correlating these measures with educational outcomes. We find that although repetition captures a significant part of uptake, pJSD outperforms repetition-based baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation. We apply our uptake measure to three different educational datasets with outcome indicators. Unlike baseline measures, pJSD correlates significantly with instruction quality in all three, providing evidence for its generalizability and for its potential to serve as an automated professional development tool for teachers.