Automatically Inferring Teachers' Geometric Content Knowledge: A Skills Based Approach

TL;DR

This paper introduces an automated, scalable method using large language models and a skills dictionary to classify teachers' Van Hiele geometric reasoning levels from open-ended responses, improving assessment efficiency.

Contribution

It develops a skills-informed classification approach for Van Hiele levels, integrating explicit reasoning skills into automated analysis, which outperforms baseline methods.

Findings

Skills-aware models significantly outperform baselines in Van Hiele classification.

The approach enables large-scale, theory-grounded assessment of teachers' geometric reasoning.

Automated classification reduces time and cost compared to manual expert analysis.

Abstract

Assessing teachers' geometric content knowledge is essential for geometry instructional quality and student learning, but difficult to scale. The Van Hiele model characterizes geometric reasoning through five hierarchical levels. Traditional Van Hiele assessment relies on manual expert analysis of open-ended responses. This process is time-consuming, costly, and prevents large-scale evaluation. This study develops an automated approach for diagnosing teachers' Van Hiele reasoning levels using large language models grounded in educational theory. Our central hypothesis is that integrating explicit skills information significantly improves Van Hiele classification. In collaboration with mathematics education researchers, we built a structured skills dictionary decomposing the Van Hiele levels into 33 fine-grained reasoning skills. Through a custom web platform, 31 pre-service teachers solved geometry problems, yielding 226 responses. Expert researchers then annotated each response with its Van Hiele level and demonstrated skills from the dictionary. Using this annotated dataset, we implemented two classification approaches: (1) retrieval-augmented generation (RAG) and (2) multi-task learning (MTL). Each approach compared a skills-aware variant incorporating the skills dictionary against a baseline without skills information. Results showed that for both methods, skills-aware variants significantly outperformed baselines across multiple evaluation metrics. This work provides the first automated approach for Van Hiele level classification from open-ended responses. It offers a scalable, theory-grounded method for assessing teachers' geometric reasoning that can enable large-scale evaluation and support adaptive, personalized teacher learning systems.