Fine-Grained Classroom Activity Detection from Audio with Neural Networks

TL;DR

This paper develops and evaluates neural network models for automatic detection of classroom activities from audio recordings, achieving state-of-the-art accuracy and robust generalization across instructors and activity types.

Contribution

It introduces deep neural architectures and feature comparisons for fine-grained classroom activity detection, advancing the automation of classroom behavior analysis.

Findings

Achieved 6.2% frame-level error rate on 4-way classification for unseen instructors.

Reduced aggregate time estimation error by 54.9% compared to baseline.

Demonstrated robust generalization across different instructors and activity classes.

Abstract

Instructors are increasingly incorporating student-centered learning techniques in their classrooms to improve learning outcomes. In addition to lecture, these class sessions involve forms of individual and group work, and greater rates of student-instructor interaction. Quantifying classroom activity is a key element of accelerating the evaluation and refinement of innovative teaching practices, but manual annotation does not scale. In this manuscript, we present advances to the young application area of automatic classroom activity detection from audio. Using a university classroom corpus with nine activity labels (e.g., "lecture," "group work," "student question"), we propose and evaluate deep fully connected, convolutional, and recurrent neural network architectures, comparing the performance of mel-filterbank, OpenSmile, and self-supervised acoustic features. We compare 9-way classification performance with 5-way and 4-way simplifications of the task and assess two types of generalization: (1) new class sessions from previously seen instructors, and (2) previously unseen instructors. We obtain strong results on the new fine-grained task and state-of-the-art on the 4-way task: our best model obtains frame-level error rates of 6.2%, 7.7% and 28.0% when generalizing to unseen instructors for the 4-way, 5-way, and 9-way classification tasks, respectively (relative reductions of 35.4%, 48.3% and 21.6% over a strong baseline). When estimating the aggregate time spent on classroom activities, our average root mean squared error is 1.64 minutes per class session, a 54.9% relative reduction over the baseline.