Multi-Task Reward Learning from Human Ratings

TL;DR

This paper introduces a multi-task reinforcement learning method that models human decision-making by integrating classification and regression tasks, improving reward inference from human ratings in RLHF.

Contribution

It proposes a novel RL approach that jointly considers multiple human decision strategies with learnable weights, capturing decision uncertainty and enhancing reward learning.

Findings

Outperforms existing rating-based RL methods

Surpasses some traditional RL approaches

Effectively models human decision-making strategies

Abstract

Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.