ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models

TL;DR

This paper introduces ERRL, a novel reinforcement learning approach that leverages expert preferences and ELO ratings to improve reward estimation and training stability in long-term tasks.

Contribution

It proposes a new reward estimation method based on ELO ratings derived from expert preferences, addressing reward design challenges in long-term RL.

Findings

ERRL outperforms state-of-the-art baselines in long-term scenarios.

The method improves training stability without fixed anchor rewards.

Expert preferences significantly influence reward estimation quality.

Abstract

Reinforcement Learning (RL) heavily relies on the careful design of the reward function. However, accurately assigning rewards to each state-action pair in Long-Term Reinforcement Learning (LTRL) tasks remains a significant challenge. As a result, RL agents are often trained under expert guidance. Inspired by the ordinal utility theory in economics, we propose a novel reward estimation algorithm: ELO-Rating based Reinforcement Learning (ERRL). This approach features two key contributions. First, it uses expert preferences over trajectories rather than cardinal rewards (utilities) to compute the ELO rating of each trajectory as its reward. Second, a new reward redistribution algorithm is introduced to alleviate training instability in the absence of a fixed anchor reward. In long-term scenarios (up to 5000 steps), where traditional RL algorithms struggle, our method outperforms several state-of-the-art baselines. Additionally, we conduct a comprehensive analysis of how expert preferences influence the results.