Gradient Q$(\sigma, \lambda)$: A Unified Algorithm with Function Approximation for Reinforcement Learning

TL;DR

This paper introduces GQ$(\sigma,\lambda)$, a unified reinforcement learning algorithm combining sampling and expectation methods with function approximation, and proves its convergence with empirical validation.

Contribution

It extends the tabular Q$(\sigma,\lambda)$ to large-scale settings using linear function approximation and provides convergence guarantees.

Findings

GQ$(\sigma,\lambda)$ outperforms traditional methods in standard domains.

The algorithm effectively combines sampling and expectation techniques.

Empirical results demonstrate improved performance over existing algorithms.

Abstract

Full-sampling (e.g., Q-learning) and pure-expectation (e.g., Expected Sarsa) algorithms are efficient and frequently used techniques in reinforcement learning. Q$(\sigma,\lambda)$ is the first approach unifies them with eligibility trace through the sampling degree $\sigma$. However, it is limited to the tabular case, for large-scale learning, the Q$(\sigma,\lambda)$ is too expensive to require a huge volume of tables to accurately storage value functions. To address above problem, we propose a GQ$(\sigma,\lambda)$ that extends tabular Q$(\sigma,\lambda)$ with linear function approximation. We prove the convergence of GQ$(\sigma,\lambda)$. Empirical results on some standard domains show that GQ$(\sigma,\lambda)$ with a combination of full-sampling with pure-expectation reach a better performance than full-sampling and pure-expectation methods.