Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards

TL;DR

This paper demonstrates that aggregating multiple objectives with different time preferences in AI requires non-Markovian rewards, challenging traditional Markovian approaches and proposing a practical aggregation scheme.

Contribution

It proves the impossibility of Markovian aggregation for diverse time preferences and introduces a non-Markovian reward scheme with minimal additional parameters.

Findings

Markovian aggregation is impossible with diverse discount factors

A practical non-Markovian aggregation scheme is proposed

Implications for designing multi-objective AI systems with varying time preferences

Abstract

As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.