Reward-Weighted Regression Converges to a Global Optimum

TL;DR

This paper proves that Reward-Weighted Regression (RWR) converges to a global optimum in a general setting without function approximation, and demonstrates R-linear convergence of the value function in finite spaces.

Contribution

It provides the first proof of global convergence for RWR and establishes R-linear convergence in finite state-action spaces.

Findings

RWR converges to a global optimum without function approximation.

In finite spaces, the state-value function converges R-linearly.

The convergence proof applies to a broad compact setting.

Abstract

Reward-Weighted Regression (RWR) belongs to a family of widely known iterative Reinforcement Learning algorithms based on the Expectation-Maximization framework. In this family, learning at each iteration consists of sampling a batch of trajectories using the current policy and fitting a new policy to maximize a return-weighted log-likelihood of actions. Although RWR is known to yield monotonic improvement of the policy under certain circumstances, whether and under which conditions RWR converges to the optimal policy have remained open questions. In this paper, we provide for the first time a proof that RWR converges to a global optimum when no function approximation is used, in a general compact setting. Furthermore, for the simpler case with finite state and action spaces we prove R-linear convergence of the state-value function to the optimum.