Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication

TL;DR

This paper demonstrates that fine-tuning pre-trained contextual embeddings like ULMFiT, BERT, and XLNet significantly improves citation content analysis in scholarly publications, outperforming traditional methods across multiple datasets.

Contribution

The study introduces a fine-tuning approach for pre-trained language models to enhance citation function and sentiment classification in scholarly texts, showing superior performance over existing baselines.

Findings

XLNet achieves up to 87.2% in citation function identification.

Our method outperforms baselines in F1 score on three datasets.

High accuracy in citation sentiment identification (over 91%).

Abstract

Citation function and citation sentiment are two essential aspects of citation content analysis (CCA), which are useful for influence analysis, the recommendation of scientific publications. However, existing studies are mostly traditional machine learning methods, although deep learning techniques have also been explored, the improvement of the performance seems not significant due to insufficient training data, which brings difficulties to applications. In this paper, we propose to fine-tune pre-trained contextual embeddings ULMFiT, BERT, and XLNet for the task. Experiments on three public datasets show that our strategy outperforms all the baselines in terms of the F1 score. For citation function identification, the XLNet model achieves 87.2%, 86.90%, and 81.6% on DFKI, UMICH, and TKDE2019 datasets respectively, while it achieves 91.72% and 91.56% on DFKI and UMICH in term of citation sentiment identification. Our method can be used to enhance the influence analysis of scholars and scholarly publications.