On finite-dimensional encoding/decoding theorems for neural operators

TL;DR

This paper extends finite-dimensional encoding/decoding theorems for neural operators to general locally convex spaces, broadening their applicability in differential equations and neural network theory.

Contribution

It proves that the approximation theorem holds for all locally convex spaces without assumptions, and characterizes when smooth mappings can be approximated in the $C^k$ topology.

Findings

The approximation theorem applies to all locally convex spaces, not just Banach spaces.

The $C^k$ approximation result holds iff the space has the approximation property.

Non-normable locally convex spaces are relevant in differential equations applications.

Abstract

Recently, versions of neural networks with infinite-dimensional affine operators inside the computational units (``neural operator'' networks) have been applied to learn solutions to differential equations. To enable practical computations, one employs finite-dimensional encoding/decoding theorems of the following kind: every continuous mapping $f$ between function spaces $E$ and $F$ is approximated in the topology of uniform convergence on compacta by continuous mappings factoring through two finite dimensional Banach spaces. Such a result is known (Kovachki et al., 2023) for $E,F$ being Banach spaces having the approximation property. We point out that the result needs no assumptions on $E,F$ whatsoever and remains true not only for all normed spaces, but for arbitrary locally convex spaces as well. At the same time, an analogous result for $C^k$-smooth mappings and the $C^k$ compact open topology, $k\geq 1$, holds if and only if the space $E$ has the approximation property. This analysis may be useful already because non-normable locally convex function spaces are common in the theory of differential equations, the main field of applications for the emerging theory.