DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models

TL;DR

This paper introduces DATE-LM, a comprehensive benchmark for evaluating data attribution methods in large language models across multiple real-world tasks, highlighting the variability and trade-offs of existing methods.

Contribution

The paper presents a unified, easy-to-use benchmark for systematic evaluation of data attribution methods tailored for LLMs, along with a large-scale comparative analysis.

Findings

No single data attribution method outperforms others across all tasks.

Data attribution methods have trade-offs with simpler baselines.

Method performance varies significantly with task-specific evaluation design.

Abstract

Data attribution methods quantify the influence of training data on model outputs and are becoming increasingly relevant for a wide range of LLM research and applications, including dataset curation, model interpretability, data valuation. However, there remain critical gaps in systematic LLM-centric evaluation of data attribution methods. To this end, we introduce DATE-LM (Data Attribution Evaluation in Language Models), a unified benchmark for evaluating data attribution methods through real-world LLM applications. DATE-LM measures attribution quality through three key tasks -- training data selection, toxicity/bias filtering, and factual attribution. Our benchmark is designed for ease of use, enabling researchers to configure and run large-scale evaluations across diverse tasks and LLM architectures. Furthermore, we use DATE-LM to conduct a large-scale evaluation of existing data attribution methods. Our findings show that no single method dominates across all tasks, data attribution methods have trade-offs with simpler baselines, and method performance is sensitive to task-specific evaluation design. Finally, we release a public leaderboard for quick comparison of methods and to facilitate community engagement, with the motivation that DATE-LM can serve as a foundation for future data attribution research in LLMs.