Attribute or Abstain: Large Language Models as Long Document Assistants

TL;DR

This paper introduces LAB, a benchmark for evaluating attribution methods in long document tasks using LLMs, revealing how different approaches impact trust and response quality.

Contribution

It provides the first long document-specific evaluation of attribution methods, comparing their effectiveness across various LLM sizes and tasks.

Findings

Citation-based attribution performs best for large, fine-tuned models.

Additional retrieval benefits small, prompted models.

Evidence quality predicts response quality for simple responses.

Abstract

LLMs can help humans working with long documents, but are known to hallucinate. Attribution can increase trust in LLM responses: The LLM provides evidence that supports its response, which enhances verifiability. Existing approaches to attribution have only been evaluated in RAG settings, where the initial retrieval confounds LLM performance. This is crucially different from the long document setting, where retrieval is not needed, but could help. Thus, a long document specific evaluation of attribution is missing. To fill this gap, we present LAB, a benchmark of 6 diverse long document tasks with attribution, and experiments with different approaches to attribution on 5 LLMs of different sizes. We find that citation, i.e. response generation and evidence extraction in one step, performs best for large and fine-tuned models, while additional retrieval can help for small, prompted models. We investigate whether the "Lost in the Middle'' phenomenon exists for attribution, but do not find this. We also find that evidence quality can predict response quality on datasets with simple responses, but not so for complex responses, as models struggle with providing evidence for complex claims.