Deep Learning without Global Optimization by Random Fourier Neural Networks

TL;DR

This paper presents a novel training method for deep neural networks using random Fourier features and MCMC sampling, avoiding traditional optimization and effectively learning complex, multiscale, high-frequency functions.

Contribution

It introduces a new training algorithm that bypasses global optimization, leveraging MCMC with complex exponential activations for efficient and interpretable deep learning.

Findings

Achieves theoretical approximation rates for residual networks.

Enables learning of multiscale and high-frequency features.

Avoids Gibbs phenomena despite sinusoidal basis functions.

Abstract

We introduce a new training algorithm for deep neural networks that utilize random complex exponential activation functions. Our approach employs a Markov Chain Monte Carlo sampling procedure to iteratively train network layers, avoiding global and gradient-based optimization while maintaining error control. It consistently attains the theoretical approximation rate for residual networks with complex exponential activation functions, determined by network complexity. Additionally, it enables efficient learning of multiscale and high-frequency features, producing interpretable parameter distributions. Despite using sinusoidal basis functions, we do not observe Gibbs phenomena in approximating discontinuous target functions.