Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale

TL;DR

Sandpiper is a system that combines human expertise and AI to analyze large-scale educational discourse data efficiently and securely, enhancing research capabilities.

Contribution

It introduces a mixed-initiative platform integrating LLMs with researcher dashboards, ensuring scalable, rigorous qualitative analysis while maintaining data privacy and reducing hallucinations.

Findings

Enables scalable qualitative analysis of educational discourse data.

Supports secure, privacy-preserving workflows with automated de-identification.

Provides continuous benchmarking of AI against human labels.

Abstract

Digital educational environments are expanding toward complex AI and human discourse, providing researchers with an abundance of data that offers deep insights into learning and instructional processes. However, traditional qualitative analysis remains a labor-intensive bottleneck, severely limiting the scale at which this research can be conducted. We present Sandpiper, a mixed-initiative system designed to serve as a bridge between high-volume conversational data and human qualitative expertise. By tightly coupling interactive researcher dashboards with agentic Large Language Model (LLM) engines, the platform enables scalable analysis without sacrificing methodological rigor. Sandpiper addresses critical barriers to AI adoption in education by implementing context-aware, automated de-identification workflows supported by secure, university-housed infrastructure to ensure data privacy. Furthermore, the system employs schema-constrained orchestration to eliminate LLM hallucinations and enforces strict adherence to qualitative codebooks. An integrated evaluations engine allows for the continuous benchmarking of AI performance against human labels, fostering an iterative approach to model refinement and validation. We propose a user study to evaluate the system's efficacy in improving research efficiency, inter-rater reliability, and researcher trust in AI-assisted qualitative workflows.