On Convergence Rate of Adaptive Multiscale Value Function Approximation For Reinforcement Learning

TL;DR

This paper introduces a multiscale, adaptive value function approximation framework for reinforcement learning using wavelet-based multiresolution analysis and tree structures, with proven convergence rates independent of basis regularity.

Contribution

It presents a novel adaptive multiscale approximation scheme for value functions in reinforcement learning, leveraging wavelet systems and tree structures for improved convergence analysis.

Findings

Convergence rate is independent of basis function regularity.

Framework effectively constructs adaptive basis functions from wavelet systems.

Provides theoretical guarantees for multiscale approximation in RL.

Abstract

In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable functions, multiresolution analysis enables us to construct a wavelet system from which the basis functions are selected adaptively, resulting in a tree structure. Furthermore, we present the convergence rate of our multiscale approximation which does not depend on the regularity of basis functions.