Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification

TL;DR

This paper introduces a multi-task learning framework with task relation learning to improve citation intention classification by fine-tuning language models on multiple datasets, enhancing generalization especially on small datasets.

Contribution

It proposes a novel multi-task learning approach with data-driven task relation learning to effectively leverage auxiliary datasets during fine-tuning of PLMs for CIC.

Findings

Improved performance on small CIC datasets by 7-11%.

Outperforms current state-of-the-art models.

Enhances generalization through auxiliary dataset integration.

Abstract

Citation intention Classification (CIC) tools classify citations by their intention (e.g., background, motivation) and assist readers in evaluating the contribution of scientific literature. Prior research has shown that pretrained language models (PLMs) such as SciBERT can achieve state-of-the-art performance on CIC benchmarks. PLMs are trained via self-supervision tasks on a large corpus of general text and can quickly adapt to CIC tasks via moderate fine-tuning on the corresponding dataset. Despite their advantages, PLMs can easily overfit small datasets during fine-tuning. In this paper, we propose a multi-task learning (MTL) framework that jointly fine-tunes PLMs on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals. We develop a data-driven task relation learning (TRL) method that controls the contribution of auxiliary datasets to avoid negative transfer and expensive hyper-parameter tuning. We conduct experiments on three CIC datasets and show that fine-tuning with additional datasets can improve the PLMs' generalization performance on the primary dataset. PLMs fine-tuned with our proposed framework outperform the current state-of-the-art models by 7% to 11% on small datasets while aligning with the best-performing model on a large dataset.