CiteCheck: Towards Accurate Citation Faithfulness Detection
Ziyao Xu, Shaohang Wei, Zhuoheng Han, Jing Jin, Zhe Yang, Xiaoguang, Li, Haochen Tan, Zhijiang Guo, Houfeng Wang

TL;DR
CiteCheck introduces the first large-scale Chinese dataset for citation faithfulness detection, enabling more accurate and cost-effective training of models, and highlights the challenge of high difficulty in test samples for current systems.
Contribution
This paper presents a novel large-scale Chinese dataset for citation faithfulness detection, created via a cost-effective two-stage manual annotation process, and demonstrates its effectiveness for training models.
Findings
Test samples are highly challenging for state-of-the-art LLMs.
Augmenting training data with LLM-generated negatives improves model performance.
The dataset facilitates research in Chinese citation faithfulness detection.
Abstract
Citation faithfulness detection is critical for enhancing retrieval-augmented generation (RAG) systems, yet large-scale Chinese datasets for this task are scarce. Existing methods face prohibitive costs due to the need for manually annotated negative samples. To address this, we introduce the first large-scale Chinese dataset CiteCheck for citation faithfulness detection, constructed via a cost-effective approach using two-stage manual annotation. This method balances positive and negative samples while significantly reducing annotation expenses. CiteCheck comprises training and test splits. Experiments demonstrate that: (1) the test samples are highly challenging, with even state-of-the-art LLMs failing to achieve high accuracy; and (2) training data augmented with LLM-generated negative samples enables smaller models to attain strong performance using parameter-efficient fine-tuning.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Text Analysis Techniques
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Byte Pair Encoding · Adam · Softmax · Dropout · Weight Decay · BART · WordPiece · Layer Normalization
