Sparse Q-learning with Mirror Descent

TL;DR

This paper introduces a novel reinforcement learning framework using mirror descent, enabling efficient sparse solutions and improved convergence through Bregman divergences and proximal-gradient methods.

Contribution

It proposes a new class of sparse mirror-descent RL algorithms that find sparse fixed points efficiently, advancing the use of Bregman divergences in reinforcement learning.

Findings

Sparse mirror-descent methods outperform previous approaches in computational efficiency.

The proposed algorithms effectively find sparse solutions in high-dimensional RL problems.

Experimental results demonstrate success in discrete and continuous MDPs.

Abstract

This paper explores a new framework for reinforcement learning based on online convex optimization, in particular mirror descent and related algorithms. Mirror descent can be viewed as an enhanced gradient method, particularly suited to minimization of convex functions in highdimensional spaces. Unlike traditional gradient methods, mirror descent undertakes gradient updates of weights in both the dual space and primal space, which are linked together using a Legendre transform. Mirror descent can be viewed as a proximal algorithm where the distance generating function used is a Bregman divergence. A new class of proximal-gradient based temporal-difference (TD) methods are presented based on different Bregman divergences, which are more powerful than regular TD learning. Examples of Bregman divergences that are studied include p-norm functions, and Mahalanobis distance based on the covariance of sample gradients. A new family of sparse mirror-descent reinforcement learning methods are proposed, which are able to find sparse fixed points of an l1-regularized Bellman equation at significantly less computational cost than previous methods based on second-order matrix methods. An experimental study of mirror-descent reinforcement learning is presented using discrete and continuous Markov decision processes.