Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making

TL;DR

This paper investigates how an agent should adapt its Pareto-optimal decision policies over time when representing multiple principals with different priors, extending classical results to more realistic scenarios.

Contribution

It generalizes Harsanyi's theorem to cases with principals having different priors, showing that weights should evolve based on observational conformity.

Findings

Harsanyi's theorem does not hold with different priors.

Weights for principals' utilities should change over time.

Implications for contract and robot design.

Abstract

It is often argued that an agent making decisions on behalf of two or more principals who have different utility functions should adopt a {\em Pareto-optimal} policy, i.e., a policy that cannot be improved upon for one agent without making sacrifices for another. A famous theorem of Harsanyi shows that, when the principals have a common prior on the outcome distributions of all policies, a Pareto-optimal policy for the agent is one that maximizes a fixed, weighted linear combination of the principals' utilities. In this paper, we show that Harsanyi's theorem does not hold for principals with different priors, and derive a more precise generalization which does hold, which constitutes our main result. In this more general case, the relative weight given to each principal's utility should evolve over time according to how well the agent's observations conform with that principal's prior. The result has implications for the design of contracts, treaties, joint ventures, and robots.