Extreme Q-Learning: MaxEnt RL without Entropy

TL;DR

This paper introduces Extreme Q-Learning, a novel approach that models the maximal Q-value directly using Extreme Value Theory, enabling effective MaxEnt RL without policy sampling, and demonstrates strong empirical performance.

Contribution

It presents the first offline MaxEnt Q-learning algorithms that do not require policy or entropy estimation, improving over prior methods by directly modeling maximal Q-values with EVT.

Findings

Outperforms prior methods by 10+ points on Franka Kitchen tasks

Achieves moderate improvements over SAC and TD3 on DM Control tasks

Demonstrates strong performance in D4RL benchmarks

Abstract

Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by \emph{10+ points} on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website at https://div99.github.io/XQL/.