Preference-based Multi-Objective Reinforcement Learning

TL;DR

This paper introduces preference-based MORL, integrating user preferences into the framework to efficiently find Pareto optimal policies without complex reward design, demonstrated through extensive experiments.

Contribution

It formalizes preference integration into MORL, providing theoretical guarantees and a practical method that outperforms traditional approaches in complex tasks.

Findings

Method performs competitively on benchmark tasks.

Outperforms oracle method using ground truth rewards.

Effective in real-world complex systems.

Abstract

Multi-objective reinforcement learning (MORL) is a structured approach for optimizing tasks with multiple objectives. However, it often relies on pre-defined reward functions, which can be hard to design for balancing conflicting goals and may lead to oversimplification. Preferences can serve as more flexible and intuitive decision-making guidance, eliminating the need for complicated reward design. This paper introduces preference-based MORL (Pb-MORL), which formalizes the integration of preferences into the MORL framework. We theoretically prove that preferences can derive policies across the entire Pareto frontier. To guide policy optimization using preferences, our method constructs a multi-objective reward model that aligns with the given preferences. We further provide theoretical proof to show that optimizing this reward model is equivalent to training the Pareto optimal policy. Extensive experiments in benchmark multi-objective tasks, a multi-energy management task, and an autonomous driving task on a multi-line highway show that our method performs competitively, surpassing the oracle method, which uses the ground truth reward function. This highlights its potential for practical applications in complex real-world systems.