Deep Neural Network Approximation of Invariant Functions through Dynamical Systems

TL;DR

This paper demonstrates that controlled equivariant dynamical systems can universally approximate permutation-invariant functions, providing theoretical foundations for designing neural networks with symmetry constraints in scientific and engineering applications.

Contribution

It establishes sufficient conditions for universal approximation of invariant functions by dynamical systems, generalizing deep residual networks with symmetry considerations.

Findings

Proves universal approximation capabilities for invariant functions using dynamical systems.

Provides guidelines for designing neural network architectures with symmetry constraints.

Connects dynamical systems theory with neural network approximation of symmetric functions.

Abstract

We study the approximation of functions which are invariant with respect to certain permutations of the input indices using flow maps of dynamical systems. Such invariant functions includes the much studied translation-invariant ones involving image tasks, but also encompasses many permutation-invariant functions that finds emerging applications in science and engineering. We prove sufficient conditions for universal approximation of these functions by a controlled equivariant dynamical system, which can be viewed as a general abstraction of deep residual networks with symmetry constraints. These results not only imply the universal approximation for a variety of commonly employed neural network architectures for symmetric function approximation, but also guide the design of architectures with approximation guarantees for applications involving new symmetry requirements.