Fair Document Valuation in LLM Summaries via Shapley Values

TL;DR

This paper proposes a Shapley value-based framework for fair document attribution in LLM summaries, introducing Cluster Shapley as an efficient approximation method that outperforms standard approaches in accuracy and fairness.

Contribution

It introduces Cluster Shapley, a novel, structure-aware approximation algorithm for fair document valuation in LLM-generated summaries, improving efficiency and accuracy.

Findings

Cluster Shapley outperforms Monte Carlo and Kernel SHAP in efficiency and accuracy.

Simple attribution rules are often highly unfair in LLM summarization.

Structure-aware approximations are promising for scalable fair content attribution.

Abstract

Large Language Models (LLMs) are increasingly used in systems that retrieve and summarize content from multiple sources, such as search engines and AI assistants. While these systems enhance user experience through coherent summaries, they obscure the individual contributions of original content creators, raising concerns about credit attribution and compensation. We address the challenge of valuing individual documents used in LLM-generated summaries by proposing a Shapley value-based framework for fair document valuation. Although theoretically appealing, exact Shapley value computation is prohibitively expensive at scale. To improve efficiency, we develop Cluster Shapley, a simple approximation algorithm that leverages semantic similarity among documents to reduce computation while maintaining attribution accuracy. Using Amazon product review data, we empirically show that off-the-shelf Shapley approximations, such as Monte Carlo sampling and Kernel SHAP, perform suboptimally in LLM settings, whereas Cluster Shapley substantially improves the efficiency-accuracy frontier. Moreover, simple attribution rules (e.g., equal or relevance-based allocation), though computationally cheap, lead to highly unfair outcomes. Together, our findings highlight the potential of structure-aware Shapley approximations tailored to LLM summarization and offer guidance for platforms seeking scalable and fair content attribution mechanisms.