Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines

TL;DR

This paper introduces a unified theoretical framework for understanding the universal approximation capabilities of both deep and shallow joint-group-equivariant neural networks using the ridgelet transform, broadening the scope of classical results.

Contribution

It provides a constructive universal approximation theorem for joint-group-equivariant machines, unifying shallow and deep network universality proofs under a common theoretical foundation.

Findings

Proves the universality of deep joint-equivariant networks.

Shows the constructive approximation properties of various network architectures.

Introduces a new depth-2 network with quadratic forms and proven universality.

Abstract

We present a constructive universal approximation theorem for learning machines equipped with joint-group-equivariant feature maps, called the joint-equivariant machines, based on the group representation theory. ``Constructive'' here indicates that the distribution of parameters is given in a closed-form expression known as the ridgelet transform. Joint-group-equivariance encompasses a broad class of feature maps that generalize classical group-equivariance. Particularly, fully-connected networks are not group-equivariant but are joint-group-equivariant. Our main theorem also unifies the universal approximation theorems for both shallow and deep networks. Until this study, the universality of deep networks has been shown in a different manner from the universality of shallow networks, but our results discuss them on common ground. Now we can understand the approximation schemes of various learning machines in a unified manner. As applications, we show the constructive universal approximation properties of four examples: depth-$n$ joint-equivariant machine, depth-$n$ fully-connected network, depth-$n$ group-convolutional network, and a new depth-$2$ network with quadratic forms whose universality has not been known.