A Theory of Regularized Markov Decision Processes

TL;DR

This paper develops a comprehensive theoretical framework for regularized Markov Decision Processes, extending existing methods by considering broader regularizers and a unified policy iteration approach, with implications for various reinforcement learning algorithms.

Contribution

It introduces a general theory that unifies and extends regularized MDPs and policy iteration, connecting to convex optimization and broadening the scope of regularizers used.

Findings

Provides a regularized Bellman operator framework

Analyzes error propagation in regularized algorithms

Connects regularized MDPs to proximal convex optimization

Abstract

Many recent successful (deep) reinforcement learning algorithms make use of regularization, generally based on entropy or Kullback-Leibler divergence. We propose a general theory of regularized Markov Decision Processes that generalizes these approaches in two directions: we consider a larger class of regularizers, and we consider the general modified policy iteration approach, encompassing both policy iteration and value iteration. The core building blocks of this theory are a notion of regularized Bellman operator and the Legendre-Fenchel transform, a classical tool of convex optimization. This approach allows for error propagation analyses of general algorithmic schemes of which (possibly variants of) classical algorithms such as Trust Region Policy Optimization, Soft Q-learning, Stochastic Actor Critic or Dynamic Policy Programming are special cases. This also draws connections to proximal convex optimization, especially to Mirror Descent.