Efficient uniform approximation using Random Vector Functional Link networks

TL;DR

This paper demonstrates that Random Vector Functional Link networks with ReLU activation can effectively approximate Lipschitz functions in the uniform norm, providing theoretical bounds on network size for desired accuracy.

Contribution

First to prove $L_$ approximation capabilities of RVFL networks with Gaussian inner weights and ReLU activations, including nonasymptotic size bounds.

Findings

RVFL with ReLU can approximate Lipschitz functions in $L_$ norm.

Provided nonasymptotic lower bounds on hidden nodes for accuracy.

Analysis based on probability theory and harmonic analysis.

Abstract

A Random Vector Functional Link (RVFL) network is a depth-2 neural network with random inner weights and biases. Only the outer weights of such an architecture are to be learned, so the learning process boils down to a linear optimization task, allowing one to sidestep the pitfalls of nonconvex optimization problems. In this paper, we prove that an RVFL with ReLU activation functions can approximate Lipschitz continuous functions in $L_\infty$ norm. To the best of our knowledge, our result is the first approximation result in $L_\infty$ norm using nice inner weights; namely, Gaussians. We give a nonasymptotic lower bound for the number of hidden-layer nodes to achieve a given accuracy with high probability, depending on, among other things, the Lipschitz constant of the target function, the desired accuracy, and the input dimension. Our method of proof is rooted in probability theory and harmonic analysis.