Regularization Guarantees Generalization in Bayesian Reinforcement Learning through Algorithmic Stability

TL;DR

This paper demonstrates that regularization in Bayesian reinforcement learning enhances policy stability and generalization, using algorithmic stability and quadratic growth conditions in non-convex MDPs.

Contribution

It introduces a novel stability analysis for regularized Bayesian RL policies without relying on convexity, leveraging recent convergence results for mirror descent in MDPs.

Findings

Regularization induces stability in Bayesian RL policies.

Stability leads to improved generalization guarantees.

Quadratic growth condition applies to regularized MDPs.

Abstract

In the Bayesian reinforcement learning (RL) setting, a prior distribution over the unknown problem parameters -- the rewards and transitions -- is assumed, and a policy that optimizes the (posterior) expected return is sought. A common approximation, which has been recently popularized as meta-RL, is to train the agent on a sample of $N$ problem instances from the prior, with the hope that for large enough $N$, good generalization behavior to an unseen test instance will be obtained. In this work, we study generalization in Bayesian RL under the probably approximately correct (PAC) framework, using the method of algorithmic stability. Our main contribution is showing that by adding regularization, the optimal policy becomes stable in an appropriate sense. Most stability results in the literature build on strong convexity of the regularized loss -- an approach that is not suitable for RL as Markov decision processes (MDPs) are not convex. Instead, building on recent results of fast convergence rates for mirror descent in regularized MDPs, we show that regularized MDPs satisfy a certain quadratic growth criterion, which is sufficient to establish stability. This result, which may be of independent interest, allows us to study the effect of regularization on generalization in the Bayesian RL setting.