Scaling Effects and Uncertainty Quantification in Neural Actor Critic Algorithms

TL;DR

This paper analyzes how different scaling schemes in neural Actor Critic algorithms affect convergence and uncertainty quantification, providing guidelines for hyperparameter selection to improve statistical robustness.

Contribution

It introduces a comprehensive statistical analysis of neural Actor Critic algorithms under various scaling regimes, extending previous convergence results to uncertainty quantification.

Findings

Variance decays as a power of network width with an exponent depending on the scaling parameter

Faster convergence observed for certain scaling choices in numerical experiments

Guidelines for hyperparameter tuning based on theoretical analysis

Abstract

We investigate the neural Actor Critic algorithm using shallow neural networks for both the Actor and Critic models. The focus of this work is twofold: first, to compare the convergence properties of the network outputs under various scaling schemes as the network width and the number of training steps tend to infinity; and second, to provide precise control of the approximation error associated with each scaling regime. Previous work has shown convergence to ordinary differential equations with random initial conditions under inverse square root scaling in the network width. In this work, we shift the focus from convergence speed alone to a more comprehensive statistical characterization of the algorithm's output, with the goal of quantifying uncertainty in neural Actor Critic methods. Specifically, we study a general inverse polynomial scaling in the network width, with an exponent treated as a tunable hyperparameter taking values strictly between one half and one. We derive an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure. To leading order, we show that the variance decays as a power of the network width, with an exponent equal to one half minus the scaling parameter, implying improved statistical robustness as the scaling parameter approaches one. Numerical experiments support this behavior and further suggest faster convergence for this choice of scaling. Finally, our analysis yields concrete guidelines for selecting algorithmic hyperparameters, including learning rates and exploration rates, as functions of the network width and the scaling parameter, ensuring provably favorable statistical behavior.