Using General Large Language Models to Classify Mathematical Documents

TL;DR

This study explores the use of general large language models to classify mathematical documents, demonstrating promising accuracy and potential improvements over existing classifications based on titles and abstracts.

Contribution

It is the first to evaluate general LLMs for classifying mathematical papers using only titles and abstracts, highlighting their potential in mathematical literature navigation.

Findings

60% of classifications matched existing labels

Half of the matches included additional classifications

In 40% of cases, LLMs suggested better classifications

Abstract

In this article we report on an initial exploration to assess the viability of using the general large language models (LLMs), recently made public, to classify mathematical documents. Automated classification would be useful from the applied perspective of improving the navigation of the literature and the more open-ended goal of identifying relations among mathematical results. The Mathematical Subject Classification MSC 2020, from MathSciNet and zbMATH, is widely used and there is a significant corpus of ground truth material in the open literature. We have evaluated the classification of preprint articles from arXiv.org according to MSC 2020. The experiment used only the title and abstract alone -- not the entire paper. Since this was early in the use of chatbots and the development of their APIs, we report here on what was carried out by hand. Of course, the automation of the process will have to follow if it is to be generally useful. We found that in about 60% of our sample the LLM produced a primary classification matching that already reported on arXiv. In about half of those instances, there were additional primary classifications that were not detected. In about 40% of our sample, the LLM suggested a different classification than what was provided. A detailed examination of these cases, however, showed that the LLM-suggested classifications were in most cases better than those provided.