Automate Knowledge Concept Tagging on Math Questions with LLMs

TL;DR

This paper investigates using Large Language Models to automate the tagging of knowledge concepts in math questions, aiming to improve efficiency and scalability in educational applications.

Contribution

It demonstrates the effectiveness of LLMs for automatic concept tagging in math questions and provides empirical insights into factors influencing their success.

Findings

LLMs show promising performance in concept tagging tasks.

Zero/few-shot learning capabilities are advantageous in educational contexts.

Key factors for successful LLM application are identified through case studies.

Abstract

Knowledge concept tagging for questions plays a crucial role in contemporary intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been conducted manually with help from pedagogical experts, as the task requires not only a strong semantic understanding of both question stems and knowledge definitions but also deep insights into connecting question-solving logic with corresponding knowledge concepts. In this paper, we explore automating the tagging task using Large Language Models (LLMs), in response to the inability of prior manual methods to meet the rapidly growing demand for concept tagging in questions posed by advanced educational applications. Moreover, the zero/few-shot learning capability of LLMs makes them well-suited for application in educational scenarios, which often face challenges in collecting large-scale, expertise-annotated datasets. By conducting extensive experiments with a variety of representative LLMs, we demonstrate that LLMs are a promising tool for concept tagging in math questions. Furthermore, through case studies examining the results from different LLMs, we draw some empirical conclusions about the key factors for success in applying LLMs to the automatic concept tagging task.