Function approximation with zonal function networks with activation functions analogous to the rectified linear unit functions

TL;DR

This paper investigates the approximation capabilities of zonal function networks on spheres using activation functions similar to ReLU, establishing their effectiveness for functions within a certain smoothness class.

Contribution

It introduces a smoothness class for functions and proves approximation properties of zonal function networks with ReLU-like activations on the sphere.

Findings

Networks can approximate functions in the smoothness class effectively.

Centers can be chosen independently of the target function.

Coefficients are linear combinations of training data.

Abstract

A zonal function (ZF) network on the $q$ dimensional sphere $\mathbb{S}^q$ is a network of the form $\mathbf{x}\mapsto \sum_{k=1}^n a_k\phi(\mathbf{x}\cdot\mathbf{x}_k)$ where $\phi :[-1,1]\to\mathbf{R}$ is the activation function, $\mathbf{x}_k\in\mathbb{S}^q$ are the centers, and $a_k\in\mathbb{R}$. While the approximation properties of such networks are well studied in the context of positive definite activation functions, recent interest in deep and shallow networks motivate the study of activation functions of the form $\phi(t)=|t|$, which are not positive definite. In this paper, we define an appropriate smoothess class and establish approximation properties of such networks for functions in this class. The centers can be chosen independently of the target function, and the coefficients are linear combinations of the training data. The constructions preserve rotational symmetries.