Generative Neural Operators of Log-Complexity Can Simultaneously Solve Infinitely Many Convex Programs

TL;DR

This paper introduces generative equilibrium neural operators (GEOs) that can efficiently approximate solutions to infinitely many convex optimization problems in Hilbert spaces, bridging the gap between theory and practice.

Contribution

The paper demonstrates that GEOs with finite-dimensional deep equilibrium layers can uniformly approximate solutions to convex optimization problems with logarithmic complexity growth, supported by theoretical analysis and practical validation.

Findings

GEOs can approximate solutions with logarithmic growth in parameters.

Theoretical bounds show uniform approximation over compact sets.

Successful application to PDEs, control, and finance problems.

Abstract

Neural operators (NOs) are a class of deep learning models designed to simultaneously solve infinitely many related problems by casting them into an infinite-dimensional space, whereon these NOs operate. A significant gap remains between theory and practice: worst-case parameter bounds from universal approximation theorems suggest that NOs may require an unrealistically large number of parameters to solve most operator learning problems, which stands in direct opposition to a slew of experimental evidence. This paper closes that gap for a specific class of {NOs}, generative {equilibrium operators} (GEOs), using (realistic) finite-dimensional deep equilibrium layers, when solving families of convex optimization problems over a separable Hilbert space $X$. Here, the inputs are smooth, convex loss functions on $X$, and outputs are the associated (approximate) solutions to the optimization problem defined by each input loss. We show that when the input losses lie in suitable infinite-dimensional compact sets, our GEO can uniformly approximate the corresponding solutions to arbitrary precision, with rank, depth, and width growing only logarithmically in the reciprocal of the approximation error. We then validate both our theoretical results and the trainability of GEOs on three applications: (1) nonlinear PDEs, (2) stochastic optimal control problems, and (3) hedging problems in mathematical finance under liquidity constraints.