Adaptive Bases for Reinforcement Learning

TL;DR

This paper introduces an adaptive basis method for reinforcement learning with function approximation, dynamically adjusting the basis to improve value function accuracy, convergence, and performance in simulated environments.

Contribution

It proposes a novel adaptive basis approach within the actor-critic framework that converges and enhances reinforcement learning performance.

Findings

Adaptive basis improves value function approximation.

Algorithms converge under the proposed framework.

Demonstrated performance gains in simulations.

Abstract

We consider the problem of reinforcement learning using function approximation, where the approximating basis can change dynamically while interacting with the environment. A motivation for such an approach is maximizing the value function fitness to the problem faced. Three errors are considered: approximation square error, Bellman residual, and projected Bellman residual. Algorithms under the actor-critic framework are presented, and shown to converge. The advantage of such an adaptive basis is demonstrated in simulations.