Knowledge Tagging with Large Language Model based Multi-Agent System

TL;DR

This paper presents a multi-agent system leveraging large language models to improve automated knowledge tagging in educational questions, especially for complex cases, demonstrating superior performance on a math dataset.

Contribution

It introduces a novel LLM-based multi-agent system that effectively handles complex knowledge tagging tasks, surpassing previous algorithms in accuracy and robustness.

Findings

Superior performance on MathKnowCT dataset

Effective handling of complex knowledge definitions

Potential for improved educational content organization

Abstract

Knowledge tagging for questions is vital in modern intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been performed by pedagogical experts, as the task demands not only a deep semantic understanding of question stems and knowledge definitions but also a strong ability to link problem-solving logic with relevant knowledge concepts. With the advent of advanced natural language processing (NLP) algorithms, such as pre-trained language models and large language models (LLMs), pioneering studies have explored automating the knowledge tagging process using various machine learning models. In this paper, we investigate the use of a multi-agent system to address the limitations of previous algorithms, particularly in handling complex cases involving intricate knowledge definitions and strict numerical constraints. By demonstrating its superior performance on the publicly available math question knowledge tagging dataset, MathKnowCT, we highlight the significant potential of an LLM-based multi-agent system in overcoming the challenges that previous methods have encountered. Finally, through an in-depth discussion of the implications of automating knowledge tagging, we underscore the promising results of deploying LLM-based algorithms in educational contexts.