Universal Approximation Theorem for Deep Q-Learning via FBSDE System

TL;DR

This paper proves a universal approximation theorem for deep Q-networks by leveraging the Bellman equation's structure and regularity properties, connecting network depth to value function refinement in control problems.

Contribution

It establishes a UAT for DQNs that emulate Bellman updates, utilizing regularity propagation and neural operators to link network architecture with dynamic programming.

Findings

Deep residual networks can approximate Bellman operators.

Network depth corresponds to value iteration steps.

Regularity of value functions is preserved through iterations.

Abstract

The approximation capabilities of Deep Q-Networks (DQNs) are commonly justified by general Universal Approximation Theorems (UATs) that do not leverage the intrinsic structural properties of the optimal Q-function, the solution to a Bellman equation. This paper establishes a UAT for a class of DQNs whose architecture is designed to emulate the iterative refinement process inherent in Bellman updates. A central element of our analysis is the propagation of regularity: while the transformation induced by a single Bellman operator application exhibits regularity, for which Backward Stochastic Differential Equations (BSDEs) theory provides analytical tools, the uniform regularity of the entire sequence of value iteration iterates--specifically, their uniform Lipschitz continuity on compact domains under standard Lipschitz assumptions on the problem data--is derived from finite-horizon dynamic programming principles. We demonstrate that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator. The resulting approximation theorem is thus intrinsically linked to the control problem's structure, offering a proof technique wherein network depth directly corresponds to iterations of value function refinement, accompanied by controlled error propagation. This perspective reveals a dynamic systems view of the network's operation on a space of value functions.