Policy Aggregation
Parand A. Alamdari, Soroush Ebadian, Ariel D. Procaccia

TL;DR
This paper addresses the challenge of combining multiple individuals' reward functions into a collective policy in AI, using social choice theory and geometric interpretations to adapt voting methods for policy aggregation.
Contribution
It introduces a novel framework that applies social choice methods to policy aggregation by interpreting preferences through occupancy polytope volumes.
Findings
Approval voting, Borda count, and other methods can be adapted for policy aggregation.
Geometric interpretation enables practical application of social choice methods.
Framework supports AI alignment with multiple stakeholders.
Abstract
We consider the challenge of AI value alignment with multiple individuals that have different reward functions and optimal policies in an underlying Markov decision process. We formalize this problem as one of policy aggregation, where the goal is to identify a desirable collective policy. We argue that an approach informed by social choice theory is especially suitable. Our key insight is that social choice methods can be reinterpreted by identifying ordinal preferences with volumes of subsets of the state-action occupancy polytope. Building on this insight, we demonstrate that a variety of methods--including approval voting, Borda count, the proportional veto core, and quantile fairness--can be practically applied to policy aggregation.
Peer Reviews
Decision·NeurIPS 2024 poster
The paper is well-written and easy to read. I believe that the problem proposed in the paper is well-motivated from alignin AI systems, and is of significant interest to research on voting rules and social choice theory. The theoretical results are solid and the proofs and/or approach are well outlined. Finally, I did not check the proofs in detail, but they appear sound. One caveat is that I am not familiar with the closely related prior work (e.g., [6]) and, so, cannot comment on the novelty o
I am not sure if the empirical results section is adding any value to this paper: it evaluates different aggregation rules, but I think this is not the focus of this work–I think the focus is to design efficient algorithms and/or prove existential results. If other reviews and the area chairs agree, my suggestion is to drop the empirical results section and use the additional space to add more exposition on the proofs. To be clear, this is no a significant concern for me.
Code & Models
Videos
Taxonomy
TopicsGame Theory and Voting Systems · Ethics and Social Impacts of AI · Mobile Crowdsensing and Crowdsourcing
