Measuring Teaching with LLMs

TL;DR

This paper introduces specialized sentence-embedding based LLMs for scalable, reliable, and valid assessment of teaching quality, outperforming human ratings and aligning with student learning outcomes.

Contribution

It presents a novel architecture using sentence-level embeddings for classroom transcripts, achieving super-human performance in teaching quality measurement.

Findings

Models achieve human-level and super-human correlation with expert ratings.

Advanced models attribute scores to lesson-level features, not just isolated utterances.

Aggregate scores correlate with student learning measures, indicating external validity.

Abstract

Objective and scalable measurement of teaching quality is a persistent challenge in education. While Large Language Models (LLMs) offer potential, general-purpose models have struggled to reliably apply complex, authentic classroom observation instruments. This paper uses custom LLMs built on sentence-level embeddings, an architecture better suited for the long-form, interpretive nature of classroom transcripts than conventional subword tokenization. We systematically evaluate five different sentence embeddings under a data-efficient training regime designed to prevent overfitting. Our results demonstrate that these specialized models can achieve human-level and even super-human performance with expert human ratings above 0.65 and surpassing the average human-human rater correlation. Further, through analysis of annotation context windows, we find that more advanced models-those better aligned with human judgments-attribute a larger share of score variation to lesson-level features rather than isolated utterances, challenging the sufficiency of single-turn annotation paradigms. Finally, to assess external validity, we find that aggregate model scores align with teacher value-added measures, indicating they are capturing features relevant to student learning. However, this trend does not hold at the individual item level, suggesting that while the models learn useful signals, they have not yet achieved full generalization. This work establishes a viable and powerful new methodology for AI-driven instructional measurement, offering a path toward providing scalable, reliable, and valid feedback for educator development.