On Convergence of Gradient Expected Sarsa($\lambda$)

TL;DR

This paper analyzes the convergence issues of Expected Sarsa(λ) with linear function approximation and introduces a new convergent gradient-based algorithm, GES(λ), with theoretical guarantees and empirical validation.

Contribution

It proposes GES(λ), a convergent gradient Expected Sarsa(λ) algorithm with proven linear convergence and a novel Lyapunov function technique for finite-time analysis.

Findings

GES(λ) converges linearly to the optimal solution.

Applying off-line estimates to Expected Sarsa(λ) is unstable off-policy.

Experimental results verify the effectiveness of GES(λ).

Abstract

We study the convergence of $\mathtt{Expected~Sarsa}(\lambda)$ with linear function approximation. We show that applying the off-line estimate (multi-step bootstrapping) to $\mathtt{Expected~Sarsa}(\lambda)$ is unstable for off-policy learning. Furthermore, based on convex-concave saddle-point framework, we propose a convergent $\mathtt{Gradient~Expected~Sarsa}(\lambda)$ ($\mathtt{GES}(\lambda)$) algorithm. The theoretical analysis shows that our $\mathtt{GES}(\lambda)$ converges to the optimal solution at a linear convergence rate, which is comparable to extensive existing state-of-the-art gradient temporal difference learning algorithms. Furthermore, we develop a Lyapunov function technique to investigate how the step-size influences finite-time performance of $\mathtt{GES}(\lambda)$, such technique of Lyapunov function can be potentially generalized to other GTD algorithms. Finally, we conduct experiments to verify the effectiveness of our $\mathtt{GES}(\lambda)$.