Unsupervised Basis Function Adaptation for Reinforcement Learning

TL;DR

This paper introduces an online adaptive state aggregation method for reinforcement learning that uses visit frequency feedback to improve value function approximation, demonstrating theoretical benefits and empirical performance gains.

Contribution

The paper presents a novel algorithm for adaptively refining approximation architectures in RL using visit frequency, with theoretical analysis and experimental validation.

Findings

Algorithm reduces value function error in tested environments.

Adaptive architecture improves RL performance over static methods.

Theoretical analysis confirms convergence and complexity bounds.

Abstract

When using reinforcement learning (RL) algorithms it is common, given a large state space, to introduce some form of approximation architecture for the value function (VF). The exact form of this architecture can have a significant effect on an agent's performance, however, and determining a suitable approximation architecture can often be a highly complex task. Consequently there is currently interest among researchers in the potential for allowing RL algorithms to adaptively generate (i.e. to learn) approximation architectures. One relatively unexplored method of adapting approximation architectures involves using feedback regarding the frequency with which an agent has visited certain states to guide which areas of the state space to approximate with greater detail. In this article we will: (a) informally discuss the potential advantages offered by such methods; (b) introduce a new algorithm based on such methods which adapts a state aggregation approximation architecture on-line and is designed for use in conjunction with SARSA; (c) provide theoretical results, in a policy evaluation setting, regarding this particular algorithm's complexity, convergence properties and potential to reduce VF error; and finally (d) test experimentally the extent to which this algorithm can improve performance given a number of different test problems. Taken together our results suggest that our algorithm (and potentially such methods more generally) can provide a versatile and computationally lightweight means of significantly boosting RL performance given suitable conditions which are commonly encountered in practice.