Inferring Lexicographically-Ordered Rewards from Preferences

TL;DR

This paper introduces a method to infer multi-objective, lexicographically-ordered reward functions from observed preferences, enabling better modeling of complex decision-making scenarios in healthcare and improving reinforcement learning policies.

Contribution

It proposes a novel approach to infer lexicographically-ordered multi-objective reward functions from preferences, addressing limitations of single reward models.

Findings

Demonstrated application in healthcare scenarios like cancer treatment and organ transplantation.

Showed that lexicographically-ordered rewards better capture complex preferences.

Improved policy recommendations using inferred multi-objective rewards.

Abstract

Modeling the preferences of agents over a set of alternatives is a principal concern in many areas. The dominant approach has been to find a single reward/utility function with the property that alternatives yielding higher rewards are preferred over alternatives yielding lower rewards. However, in many settings, preferences are based on multiple, often competing, objectives; a single reward function is not adequate to represent such preferences. This paper proposes a method for inferring multi-objective reward-based representations of an agent's observed preferences. We model the agent's priorities over different objectives as entering lexicographically, so that objectives with lower priorities matter only when the agent is indifferent with respect to objectives with higher priorities. We offer two example applications in healthcare, one inspired by cancer treatment, the other inspired by organ transplantation, to illustrate how the lexicographically-ordered rewards we learn can provide a better understanding of a decision-maker's preferences and help improve policies when used in reinforcement learning.