On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

TL;DR

This paper provides a theoretical analysis of deep neural networks' ability to approximate invariant functions, demonstrating their universal approximation capability and efficiency across various architectures, with practical applications and empirical validation.

Contribution

It offers the first theoretical proof that deep neural networks can universally approximate invariant functions and shows their efficiency across multiple architectures with practical applications.

Findings

Invariant functions can be universally approximated by deep neural networks.

Various neural network models can asymptotically approximate invariant functions efficiently.

Empirical results validate the effectiveness of the proposed approximation methods.

Abstract

Recent years have witnessed a hot wave of deep neural networks in various domains; however, it is not yet well understood theoretically. A theoretical characterization of deep neural networks should point out their approximation ability and complexity, i.e., showing which architecture and size are sufficient to handle the concerned tasks. This work takes one step on this direction by theoretically studying the approximation and complexity of deep neural networks to invariant functions. We first prove that the invariant functions can be universally approximated by deep neural networks. Then we show that a broad range of invariant functions can be asymptotically approximated by various types of neural network models that includes the complex-valued neural networks, convolutional neural networks, and Bayesian neural networks using a polynomial number of parameters or optimization iterations. We also provide a feasible application that connects the parameter estimation and forecasting of high-resolution signals with our theoretical conclusions. The empirical results obtained on simulation experiments demonstrate the effectiveness of our method.