Generalized Priority-Aware Shapley Value

TL;DR

This paper introduces the generalized priority-aware Shapley value (GPASV), a flexible valuation method for complex priority graphs in machine learning, extending classical models to handle cyclic and weighted preferences.

Contribution

The paper develops GPASV, a novel axiomatic and computational framework for priority-aware valuation on arbitrary directed weighted graphs, addressing real-world cyclic preferences.

Findings

GPASV effectively models cyclic and weighted priority graphs.

Different priority balances significantly affect valuation outcomes.

Application to LLM ensemble valuation demonstrates practical utility.

Abstract

Shapley value and its priority-aware extensions are widely used for valuation in machine learning, but existing methods require pairwise priority to be binary and acyclic, a restriction spectacularly violated in real-data examples such as aggregated human preferences and multi-criterion comparisons. We introduce the generalized priority-aware Shapley value (GPASV), a random order value defined on arbitrary directed weighted priority graphs, in which pairwise edges penalize rather than forbid order violations. GPASV covers a range of classical models as boundary cases. We establish GPASV through an axiomatic characterization, develop the associated computational methods, and introduce a priority sweeping diagnostic extending PASV's. We apply GPASV to LLM ensemble valuation on the cyclic Chatbot Arena preference graph, illustrating that priority-aware valuation is not a one-button operation: different balances of pairwise graph priority versus individual soft priority produce substantively different valuations of the same data.