On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts

TL;DR

This paper evaluates the effectiveness of large language models in automatically categorizing scientific texts, demonstrating that recent LLMs outperform traditional models in accuracy for hierarchical classification tasks.

Contribution

The study provides empirical evidence that recent LLMs, like Meta Llama 3.1, significantly improve scientific text classification accuracy over traditional models such as BERT.

Findings

LLMs achieve up to 0.82 accuracy on the FORC dataset.

Recent LLMs outperform BERT by up to 0.08 in accuracy.

LLMs show promise for enhancing research information systems.

Abstract

The rapid advancement of Large Language Models (LLMs) has led to a multitude of application opportunities. One traditional task for Information Retrieval systems is the summarization and classification of texts, both of which are important for supporting humans in navigating large literature bodies as they e.g. exist with scientific publications. Due to this rapidly growing body of scientific knowledge, recent research has been aiming at building research information systems that not only offer traditional keyword search capabilities, but also novel features such as the automatic detection of research areas that are present at knowledge intensive organizations in academia and industry. To facilitate this idea, we present the results obtained from evaluating a variety of LLMs in their ability to sort scientific publications into hierarchical classifications systems. Using the FORC dataset as ground truth data, we have found that recent LLMs (such as Meta Llama 3.1) are able to reach an accuracy of up to 0.82, which is up to 0.08 better than traditional BERT models.