Content Knowledge Identification with Multi-Agent Large Language Models (LLMs)

TL;DR

This paper introduces LLMAgent-CK, a multi-agent LLM framework that automatically assesses teachers' mathematical content knowledge from responses, enhancing asynchronous professional development with minimal human annotation.

Contribution

The paper presents a novel multi-agent LLM-based framework for automatic CK identification that addresses data scarcity and interpretability issues in teacher PD systems.

Findings

Achieves promising CK identification performance on real-world dataset

Demonstrates effective human-like discussions in multi-agent framework

Shows potential for scalable, annotation-free teacher assessment

Abstract

Teachers' mathematical content knowledge (CK) is of vital importance and need in teacher professional development (PD) programs. Computer-aided asynchronous PD systems are the most recent proposed PD techniques, which aim to help teachers improve their PD equally with fewer concerns about costs and limitations of time or location. However, current automatic CK identification methods, which serve as one of the core techniques of asynchronous PD systems, face challenges such as diversity of user responses, scarcity of high-quality annotated data, and low interpretability of the predictions. To tackle these challenges, we propose a Multi-Agent LLMs-based framework, LLMAgent-CK, to assess the user responses' coverage of identified CK learning goals without human annotations. By taking advantage of multi-agent LLMs in strong generalization ability and human-like discussions, our proposed LLMAgent-CK presents promising CK identifying performance on a real-world mathematical CK dataset MaCKT. Moreover, our case studies further demonstrate the working of the multi-agent framework.