Universal approximation properties of shallow quadratic neural networks

TL;DR

This paper demonstrates that shallow quadratic neural networks are universal approximators, often requiring fewer neurons than standard networks, and shows their effectiveness in clustering tasks like MNIST.

Contribution

It introduces and proves the universality of shallow quadratic neural networks and compares their efficiency to standard networks in approximation and clustering tasks.

Findings

Quadratic neural networks require fewer neurons for approximation.

They achieve comparable or better clustering performance on MNIST.

Convergence rates are established using wavelet and statistical learning theory.

Abstract

In this paper we study shallow neural network functions which are linear combinations of compositions of activation and quadratic functions, replacing standard affine linear functions, often called neurons. We show the universality of this approximation and prove convergence rates results based on the theory of wavelets and statistical learning. We show for simple test cases that this ansatz requires a smaller numbers of neurons than standard affine linear neural networks. Moreover, we investigate the efficiency of this approach for clustering tasks with the MNIST data set. Similar observations are made when comparing deep (multi-layer) networks.