Minimax Optimal Variance-Aware Regret Bounds for Multinomial Logistic MDPs

TL;DR

This paper introduces a new regret bound for reinforcement learning in multinomial logistic MDPs that adapts to problem structure, achieving minimax optimality and improving previous bounds for certain cases.

Contribution

The authors propose a variance-aware algorithm with regret bounds that adapt to the normalized variance of the value function, improving upon existing bounds and establishing minimax optimality.

Findings

The new regret bound depends on a problem-dependent variance measure.

For KL-constrained robust MDPs, the bound reduces horizon dependence by a factor of H.

The paper proves a matching lower bound, establishing minimax optimality.

Abstract

We study reinforcement learning for episodic Markov Decision Processes (MDPs) whose transitions are modelled by a multinomial logistic (MNL) model. Existing algorithms for MNL mixture MDPs yield a regret of $\smash{\tilde{O}(dH^2\sqrt{T})}$ (Li et al., 2024), where $d$ is the feature dimension, $H$ the episode length, and $T$ the number of episodes. Inspired by the logistic bandit literature (Abeille et al., 2021; Faury et al., 2022; Boudart et al., 2026), we introduce a problem-dependent constant $\bar\sigma\_T \leq 1/2$, measuring the normalised average variance of the optimal downstream value function along the learner's trajectory. We propose an algorithm achieving a regret of $\smash{\tilde{O}(dH^2\bar\sigma\_T\sqrt{T})}$, which recovers the existing bound in the worst case and improves upon it for structured MDPs. For instance, for KL-constrained robust MDPs, $\bar\sigma\_T = O(H^{-1})$, reducing the horizon dependence by a factor $H$. We further establish a matching $\smash{\Omega(dH^2\bar\sigma\_T\sqrt{T})}$ lower bound, proving minimax optimality (up to logarithmic factors) and fully characterising the regret complexity of MNL mixture MDPs for the first time.