Can neural operators always be continuously discretized?

TL;DR

This paper investigates the discretization of neural operators, revealing fundamental limitations for diffeomorphisms and proposing strongly monotone neural operators as a robust alternative with guaranteed discretization invariance.

Contribution

The paper introduces strongly monotone neural operators as a new class that can be discretized reliably, overcoming limitations of traditional diffeomorphic neural operators.

Findings

Diffeomorphisms between infinite-dimensional Hilbert spaces may not be approximable by finite-dimensional diffeomorphisms.

Strongly monotone neural operators can be discretized with guarantees of convergence and invariance.

Any bilipschitz neural operator can be decomposed into strongly monotone components plus an isometry.

Abstract

We consider the problem of discretization of neural operators between Hilbert spaces in a general framework including skip connections. We focus on bijective neural operators through the lens of diffeomorphisms in infinite dimensions. Framed using category theory, we give a no-go theorem that shows that diffeomorphisms between Hilbert spaces or Hilbert manifolds may not admit any continuous approximations by diffeomorphisms on finite-dimensional spaces, even if the approximations are nonlinear. The natural way out is the introduction of strongly monotone diffeomorphisms and layerwise strongly monotone neural operators which have continuous approximations by strongly monotone diffeomorphisms on finite-dimensional spaces. For these, one can guarantee discretization invariance, while ensuring that finite-dimensional approximations converge not only as sequences of functions, but that their representations converge in a suitable sense as well. Finally, we show that bilipschitz neural operators may always be written in the form of an alternating composition of strongly monotone neural operators, plus a simple isometry. Thus we realize a rigorous platform for discretization of a generalization of a neural operator. We also show that neural operators of this type may be approximated through the composition of finite-rank residual neural operators, where each block is strongly monotone, and may be inverted locally via iteration. We conclude by providing a quantitative approximation result for the discretization of general bilipschitz neural operators.