Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Study

TL;DR

This study evaluates the performance of various large language models in scientific text classification, demonstrating that domain-specific models like SciBERT outperform general-purpose models, highlighting the importance of domain adaptation.

Contribution

It provides a comparative analysis of fine-tuning four LLMs on scientific datasets, emphasizing the superiority of domain-specific models for specialized text classification tasks.

Findings

SciBERT outperforms other models in accuracy

Domain-specific models excel in scientific text classification

LLMs show advantages over traditional deep learning models in domain tasks

Abstract

The exponential growth of online textual content across diverse domains has necessitated advanced methods for automated text classification. Large Language Models (LLMs) based on transformer architectures have shown significant success in this area, particularly in natural language processing (NLP) tasks. However, general-purpose LLMs often struggle with domain-specific content, such as scientific texts, due to unique challenges like specialized vocabulary and imbalanced data. In this study, we fine-tune four state-of-the-art LLMs BERT, SciBERT, BioBERT, and BlueBERT on three datasets derived from the WoS-46985 dataset to evaluate their performance in scientific text classification. Our experiments reveal that domain-specific models, particularly SciBERT, consistently outperform general-purpose models in both abstract-based and keyword-based classification tasks. Additionally, we compare our achieved results with those reported in the literature for deep learning models, further highlighting the advantages of LLMs, especially when utilized in specific domains. The findings emphasize the importance of domain-specific adaptations for LLMs to enhance their effectiveness in specialized text classification tasks.