Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

TL;DR

This paper introduces a hierarchical, multi-stage framework for improving the reliability of LLM-based classroom discourse annotation by mimicking human annotation workflows and explicitly modeling computational tradeoffs.

Contribution

It presents a novel multi-stage orchestration framework that enhances annotation reliability and validity for educational data using cost-aware LLM workflows.

Findings

Hierarchical framework improves annotation accuracy over single-pass models

Self-verification reduces errors in initial LLM annotations

Disagreement adjudication aligns model outputs with human expert standards

Abstract

Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learning artifacts. Their ability to rapidly summarize instructional interactions and assign rubric-aligned labels has fueled optimism about reducing the cost and time associated with expert human annotation. However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs that require contextual, pedagogical, or normative judgment, such as instructional intent or discourse moves. This tension between scale and validity sits at the core of contemporary education data science. In this work, we present and empirically evaluate a hierarchical, cost-aware orchestration framework for LLM-based annotation that improves reliability while explicitly modeling computational tradeoffs. Rather than treating annotation as a one-shot prediction problem, we conceptualize it as a multi-stage epistemic process comprising (1) an unverified single-pass annotation stage, in which models independently assign labels based on the rubric; (2) a self-verification stage, in which each model audits its own output against rubric definitions and revises its label if inconsistencies are detected; and (3) a disagreement-centric adjudication stage, in which an independent adjudicator model examines the verified labels and justifications and determines a final label in accordance with the rubric. This structure mirrors established human annotation workflows in educational research, where initial coding is followed by self-checking and expert resolution of disagreements.