Judging the Judges: Human Validation of Multi-LLM Evaluation for High-Quality K--12 Science Instructional Materials

TL;DR

This study evaluates how human experts interpret AI-generated assessments of K--12 science instructional materials, aiming to inform the development of a GenAI tool for designing high-quality, standards-aligned educational content.

Contribution

It analyzes expert review patterns of LLM evaluations to identify strengths, gaps, and nuances, guiding the creation of a domain-specific GenAI instructional design agent.

Findings

Experts identify key reasoning strengths and gaps in LLM evaluations.

Patterns of agreement and disagreement reveal contextual nuances in AI judgments.

Insights will inform the development of a specialized GenAI tool for instructional design.

Abstract

Designing high-quality, standards-aligned instructional materials for K--12 science is time-consuming and expertise-intensive. This study examines what human experts notice when reviewing AI-generated evaluations of such materials, aiming to translate their insights into design principles for a future GenAI-based instructional material design agent. We intentionally selected 12 high-quality curriculum units across life, physical, and earth sciences from validated programs such as OpenSciEd and Multiple Literacies in Project-based Learning. Using the EQuIP rubric with 9 evaluation items, we prompted GPT-4o, Claude, and Gemini to produce numerical ratings and written rationales for each unit, generating 648 evaluation outputs. Two science education experts independently reviewed all outputs, marking agreement (1) or disagreement (0) for both scores and rationales, and offering qualitative reflections on AI reasoning. This process surfaces patterns in where LLM judgments align with or diverge from expert perspectives, revealing reasoning strengths, gaps, and contextual nuances. These insights will directly inform the development of a domain-specific GenAI agent to support the design of high-quality instructional materials in K--12 science education.