Optimal Approximation with Sparse Neural Networks and Applications

TL;DR

This paper develops a theoretical framework for understanding the approximation capabilities of deep sparse neural networks, linking them to classical mathematical concepts like representation systems and applying these ideas to practical function approximation tasks.

Contribution

It introduces a fundamental bound theorem connecting neural network approximation to intrinsic function class properties and transfers approximation theories from representation systems to neural networks.

Findings

Neural networks can approximate B-spline functions effectively.

A bound theorem relates neural network approximation to function class complexity.

Application of rate-distortion theory to analyze neural network approximation of cartoon-like functions.

Abstract

We use deep sparsely connected neural networks to measure the complexity of a function class in $L^2(\mathbb R^d)$ by restricting connectivity and memory requirement for storing the neural networks. We also introduce representation system - a countable collection of functions to guide neural networks, since approximation theory with representation system has been well developed in Mathematics. We then prove the fundamental bound theorem, implying a quantity intrinsic to the function class itself can give information about the approximation ability of neural networks and representation system. We also provides a method for transferring existing theories about approximation by representation systems to that of neural networks, greatly amplifying the practical values of neural networks. Finally, we use neural networks to approximate B-spline functions, which are used to generate the B-spline curves. Then, we analyse the complexity of a class called $\beta$ cartoon-like functions using rate-distortion theory and wedgelets construction.