Efficient Few-shot Learning for Multi-label Classification of Scientific Documents with Many Classes

TL;DR

FusionSent is a novel approach that fine-tunes and fuses sentence embedding models for efficient, few-shot multi-label classification of scientific documents, significantly outperforming baselines on new and existing datasets.

Contribution

The paper introduces FusionSent, a fusion-based sentence embedding fine-tuning method for few-shot classification of scientific documents with many classes, along with a new large-scale dataset.

Findings

FusionSent outperforms strong baselines by 6 F1 points.

The method is efficient and effective for label-scarce scenarios.

A new dataset with 203,961 articles and 130 classes is provided.

Abstract

Scientific document classification is a critical task and often involves many classes. However, collecting human-labeled data for many classes is expensive and usually leads to label-scarce scenarios. Moreover, recent work has shown that sentence embedding model fine-tuning for few-shot classification is efficient, robust, and effective. In this work, we propose FusionSent (Fusion-based Sentence Embedding Fine-tuning), an efficient and prompt-free approach for few-shot classification of scientific documents with many classes. FusionSent uses available training examples and their respective label texts to contrastively fine-tune two different sentence embedding models. Afterward, the parameters of both fine-tuned models are fused to combine the complementary knowledge from the separate fine-tuning steps into a single model. Finally, the resulting sentence embedding model is frozen to embed the training instances, which are then used as input features to train a classification head. Our experiments show that FusionSent significantly outperforms strong baselines by an average of $6.0$ $F_{1}$ points across multiple scientific document classification datasets. In addition, we introduce a new dataset for multi-label classification of scientific documents, which contains 203,961 scientific articles and 130 classes from the arXiv category taxonomy. Code and data are available at https://github.com/sebischair/FusionSent.