Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedback

TL;DR

This paper introduces a framework for personalized multi-objective decision making that learns user preferences through comparative feedback, enabling the computation of near-optimal policies tailored to individual priorities.

Contribution

It proposes a novel approach to incorporate user preferences into multi-objective RL via comparison-based feedback models and efficient algorithms for preference elicitation.

Findings

Effective algorithms for preference elicitation with few comparison queries

Ability to learn personalized policies in multi-objective settings

Framework accommodates different types of user feedback

Abstract

In classic reinforcement learning (RL) and decision making problems, policies are evaluated with respect to a scalar reward function, and all optimal policies are the same with regards to their expected return. However, many real-world problems involve balancing multiple, sometimes conflicting, objectives whose relative priority will vary according to the preferences of each user. Consequently, a policy that is optimal for one user might be sub-optimal for another. In this work, we propose a multi-objective decision making framework that accommodates different user preferences over objectives, where preferences are learned via policy comparisons. Our model consists of a Markov decision process with a vector-valued reward function, with each user having an unknown preference vector that expresses the relative importance of each objective. The goal is to efficiently compute a near-optimal policy for a given user. We consider two user feedback models. We first address the case where a user is provided with two policies and returns their preferred policy as feedback. We then move to a different user feedback model, where a user is instead provided with two small weighted sets of representative trajectories and selects the preferred one. In both cases, we suggest an algorithm that finds a nearly optimal policy for the user using a small number of comparison queries.