Attribution in Scientific Literature: New Benchmark and Methods

TL;DR

This paper introduces REASONS, a new benchmark dataset for scientific citation attribution, and evaluates LLMs' performance and hallucination issues, proposing metadata augmentation and retrieval techniques to improve reliability.

Contribution

The paper presents REASONS, a comprehensive dataset with annotation across scientific domains, and evaluates methods to reduce hallucinations in LLM-based citation tasks.

Findings

Top-tier LLMs achieve high sentence attribution accuracy.

Metadata augmentation reduces hallucination rates across tasks.

Retrieval-augmented generation improves indirect query performance.

Abstract

Large language models (LLMs) present a promising yet challenging frontier for automated source citation in scientific communication. Previous approaches to citation generation have been limited by citation ambiguity and LLM overgeneralization. We introduce REASONS, a novel dataset with sentence-level annotations across 12 scientific domains from arXiv. Our evaluation framework covers two key citation scenarios: indirect queries (matching sentences to paper titles) and direct queries (author attribution), both enhanced with contextual metadata. We conduct extensive experiments with models such as GPT-O1, GPT-4O, GPT-3.5, DeepSeek, and other smaller models like Perplexity AI (7B). While top-tier LLMs achieve high performance in sentence attribution, they struggle with high hallucination rates, a key metric for scientific reliability. Our metadata-augmented approach reduces hallucination rates across all tasks, offering a promising direction for improvement. Retrieval-augmented generation (RAG) with Mistral improves performance in indirect queries, reducing hallucination rates by 42% and maintaining competitive precision with larger models. However, adversarial testing highlights challenges in linking paper titles to abstracts, revealing fundamental limitations in current LLMs. REASONS provides a challenging benchmark for developing reliable and trustworthy LLMs in scientific applications