Going Beyond Linear RL: Sample Efficient Neural Function Approximation

TL;DR

This paper explores nonlinear neural network approaches in deep reinforcement learning, providing efficient algorithms with improved sample complexity over traditional linear methods, especially for two-layer neural networks.

Contribution

It introduces the first efficient algorithms for nonlinear neural network Q-function approximation in RL, with theoretical guarantees under various assumptions.

Findings

Algorithms are computationally and statistically efficient in the generative model setting.

Sample complexity scales linearly with the algebraic dimension under realizability.

Results outperform linear and eluder dimension-based methods.

Abstract

Deep Reinforcement Learning (RL) powered by neural net approximation of the Q function has had enormous empirical success. While the theory of RL has traditionally focused on linear function approximation (or eluder dimension) approaches, little is known about nonlinear RL with neural net approximations of the Q functions. This is the focus of this work, where we study function approximation with two-layer neural networks (considering both ReLU and polynomial activation functions). Our first result is a computationally and statistically efficient algorithm in the generative model setting under completeness for two-layer neural networks. Our second result considers this setting but under only realizability of the neural net function class. Here, assuming deterministic dynamics, the sample complexity scales linearly in the algebraic dimension. In all cases, our results significantly improve upon what can be attained with linear (or eluder dimension) methods.