Optimization for Neural Operators can Benefit from Width

TL;DR

This paper establishes convergence guarantees for gradient descent in neural operators like DONs and FNOs, showing that wider networks improve optimization, supported by theoretical analysis and empirical experiments.

Contribution

The paper introduces a unified framework for analyzing GD convergence in neural operators and demonstrates that wider networks enhance optimization performance.

Findings

GD convergence is guaranteed under RSC and smoothness conditions.

Wider neural operators lead to better optimization convergence.

Empirical results support the theoretical findings.

Abstract

Neural Operators that directly learn mappings between function spaces, such as Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), have received considerable attention. Despite the universal approximation guarantees for DONs and FNOs, there is currently no optimization convergence guarantee for learning such networks using gradient descent (GD). In this paper, we address this open problem by presenting a unified framework for optimization based on GD and applying it to establish convergence guarantees for both DONs and FNOs. In particular, we show that the losses associated with both of these neural operators satisfy two conditions -- restricted strong convexity (RSC) and smoothness -- that guarantee a decrease on their loss values due to GD. Remarkably, these two conditions are satisfied for each neural operator due to different reasons associated with the architectural differences of the respective models. One takeaway that emerges from the theory is that wider networks should lead to better optimization convergence for both DONs and FNOs. We present empirical results on canonical operator learning problems to support our theoretical results.