Statistical analysis of Inverse Entropy-regularized Reinforcement Learning

TL;DR

This paper introduces a statistical framework for inverse entropy-regularized reinforcement learning that uniquely recovers the reward function by combining entropy regularization with least-squares reconstruction, providing theoretical guarantees.

Contribution

It develops a novel statistical approach that resolves reward ambiguity in IRL using entropy regularization and establishes minimax optimal convergence rates.

Findings

Provides high-probability bounds for policy estimation error.

Derives non-asymptotic minimax optimal rates for reward recovery.

Bridges IRL with modern statistical learning theory.

Abstract

Inverse reinforcement learning aims to infer the reward function that explains expert behavior observed through trajectories of state--action pairs. A long-standing difficulty in classical IRL is the non-uniqueness of the recovered reward: many reward functions can induce the same optimal policy, rendering the inverse problem ill-posed. In this paper, we develop a statistical framework for Inverse Entropy-regularized Reinforcement Learning that resolves this ambiguity by combining entropy regularization with a least-squares reconstruction of the reward from the soft Bellman residual. This combination yields a unique and well-defined so-called least-squares reward consistent with the expert policy. We model the expert demonstrations as a Markov chain with the invariant distribution defined by an unknown expert policy $\pi^\star$ and estimate the policy by a penalized maximum-likelihood procedure over a class of conditional distributions on the action space. We establish high-probability bounds for the excess Kullback--Leibler divergence between the estimated policy and the expert policy, accounting for statistical complexity through covering numbers of the policy class. These results lead to non-asymptotic minimax optimal convergence rates for the least-squares reward function, revealing the interplay between smoothing (entropy regularization), model complexity, and sample size. Our analysis bridges the gap between behavior cloning, inverse reinforcement learning, and modern statistical learning theory.