Asymptotics of Reinforcement Learning with Neural Networks

TL;DR

This paper establishes the asymptotic behavior of single-layer neural networks trained with Q-learning, showing convergence to solutions of the Bellman equation and analyzing their limiting dynamics.

Contribution

It provides the first rigorous analysis of the asymptotics of neural networks in reinforcement learning, connecting neural training dynamics to classical control solutions.

Findings

Neural networks converge to a differential equation as size and training steps grow.

The differential equation has a unique stationary solution solving the Bellman equation.

Analysis reveals the limiting behavior of neural networks trained with stochastic gradient descent.

Abstract

We prove that a single-layer neural network trained with the Q-learning algorithm converges in distribution to a random ordinary differential equation as the size of the model and the number of training steps become large. Analysis of the limit differential equation shows that it has a unique stationary solution which is the solution of the Bellman equation, thus giving the optimal control for the problem. In addition, we study the convergence of the limit differential equation to the stationary solution. As a by-product of our analysis, we obtain the limiting behavior of single-layer neural networks when trained on i.i.d. data with stochastic gradient descent under the widely-used Xavier initialization.