Adapting the Function Approximation Architecture in Online Reinforcement Learning

TL;DR

This paper introduces an adaptive function approximation architecture for online reinforcement learning that efficiently discovers useful features in high-dimensional, noisy observations, improving performance over non-adaptive methods.

Contribution

It proposes a novel online RL prediction algorithm with an adaptive architecture tailored for unknown observation structures, enhancing scalability and effectiveness.

Findings

Outperforms non-adaptive baseline architectures

Approaches the performance of architectures with side-channel information

Effective in high-dimensional, stochastic spatial domains

Abstract

The performance of a reinforcement learning (RL) system depends on the computational architecture used to approximate a value function. Deep learning methods provide both optimization techniques and architectures for approximating nonlinear functions from noisy, high-dimensional observations. However, prevailing optimization techniques are not designed for strictly-incremental online updates. Nor are standard architectures designed for observations with an a priori unknown structure: for example, light sensors randomly dispersed in space. This paper proposes an online RL prediction algorithm with an adaptive architecture that efficiently finds useful nonlinear features. The algorithm is evaluated in a spatial domain with high-dimensional, stochastic observations. The algorithm outperforms non-adaptive baseline architectures and approaches the performance of an architecture given side-channel information. These results are a step towards scalable RL algorithms for more general problems, where the observation structure is not available.