Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences

TL;DR

This paper introduces Multi-Type Preference Learning (MTPL), a novel approach in preference-based reinforcement learning that incorporates equal preferences, leading to improved understanding and efficiency in learning from human feedback.

Contribution

The paper proposes the Equal Preference Learning Task and MTPL, enabling simultaneous learning from equal and explicit preferences, which enhances feedback utilization in PBRL.

Findings

MTPL improves learning efficiency across multiple tasks.

Incorporating equal preferences enhances agent understanding.

Experimental results outperform existing baselines.

Abstract

Preference-Based reinforcement learning (PBRL) learns directly from the preferences of human teachers regarding agent behaviors without needing meticulously designed reward functions. However, existing PBRL methods often learn primarily from explicit preferences, neglecting the possibility that teachers may choose equal preferences. This neglect may hinder the understanding of the agent regarding the task perspective of the teacher, leading to the loss of important information. To address this issue, we introduce the Equal Preference Learning Task, which optimizes the neural network by promoting similar reward predictions when the behaviors of two agents are labeled as equal preferences. Building on this task, we propose a novel PBRL method, Multi-Type Preference Learning (MTPL), which allows simultaneous learning from equal preferences while leveraging existing methods for learning from explicit preferences. To validate our approach, we design experiments applying MTPL to four existing state-of-the-art baselines across ten locomotion and robotic manipulation tasks in the DeepMind Control Suite. The experimental results indicate that simultaneous learning from both equal and explicit preferences enables the PBRL method to more comprehensively understand the feedback from teachers, thereby enhancing feedback efficiency. Project page: \url{https://github.com/FeiCuiLengMMbb/paper_MTPL}