Can LLMs Predict Citation Intent? An Experimental Analysis of In-context Learning and Fine-tuning on Open LLMs

TL;DR

This paper explores how open large language models can predict citation intent using in-context learning and fine-tuning, showing that general-purpose models can outperform domain-specific models with minimal data.

Contribution

It demonstrates the effectiveness of open LLMs in citation intent prediction and highlights the benefits of fine-tuning over in-context learning for improved performance.

Findings

Fine-tuning improves F1-score by 8% on SciCite dataset.

Top-performing model identified through extensive experiments.

Open evaluation framework and models released for future research.

Abstract

This work investigates the ability of open Large Language Models (LLMs) to predict citation intent through in-context learning and fine-tuning. Unlike traditional approaches relying on domain-specific pre-trained models like SciBERT, we demonstrate that general-purpose LLMs can be adapted to this task with minimal task-specific data. We evaluate twelve model variations across five prominent open LLM families using zero-, one-, few-, and many-shot prompting. Our experimental study identifies the top-performing model and prompting parameters through extensive in-context learning experiments. We then demonstrate the significant impact of task-specific adaptation by fine-tuning this model, achieving a relative F1-score improvement of 8% on the SciCite dataset and 4.3% on the ACL-ARC dataset compared to the instruction-tuned baseline. These findings provide valuable insights for model selection and prompt engineering. Additionally, we make our end-to-end evaluation framework and models openly available for future use.