ACM-CR: A Manually Annotated Test Collection for Citation Recommendation

TL;DR

This paper introduces ACM-CR, a high-quality, manually annotated test collection for citation recommendation, aiming to improve evaluation reliability over existing noisy datasets.

Contribution

It presents a new manually annotated dataset for citation recommendation and evaluates baseline models, facilitating more accurate future research.

Findings

Baseline models show moderate effectiveness on ACM-CR

Manual annotation improves dataset reliability

Provides open access to dataset and code

Abstract

Citation recommendation is intended to assist researchers in the process of searching for relevant papers to cite by recommending appropriate citations for a given input text. Existing test collections for this task are noisy and unreliable since they are built automatically from parsed PDF papers. In this paper, we present our ongoing effort at creating a publicly available, manually annotated test collection for citation recommendation. We also conduct a series of experiments to evaluate the effectiveness of content-based baseline models on the test collection, providing results for future work to improve upon. Our test collection and code to replicate experiments are available at https://github.com/boudinfl/acm-cr