Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel

TL;DR

This paper explores multilayer neural fields with correlated random weights, demonstrating their robustness and slightly improved generalization over traditional deep networks, supported by numerical experiments aligned with neural tangent kernel theory.

Contribution

It introduces a novel neural field model with correlated weights, analyzing its training landscape and generalization properties through numerical experiments and neural tangent kernel insights.

Findings

Neural fields exhibit the existence of global minima in their parameter space.

The neural fields are more robust to input noise compared to conventional models.

They show slightly better generalization performance in experiments.

Abstract

A biological neural network in the cortex forms a neural field. Neurons in the field have their own receptive fields, and connection weights between two neurons are random but highly correlated when they are in close proximity in receptive fields. In this paper, we investigate such neural fields in a multilayer architecture to investigate the supervised learning of the fields. We empirically compare the performances of our field model with those of randomly connected deep networks. The behavior of a randomly connected network is investigated on the basis of the key idea of the neural tangent kernel regime, a recent development in the machine learning theory of over-parameterized networks; for most randomly connected neural networks, it is shown that global minima always exist in their small neighborhoods. We numerically show that this claim also holds for our neural fields. In more detail, our model has two structures: i) each neuron in a field has a continuously distributed receptive field, and ii) the initial connection weights are random but not independent, having correlations when the positions of neurons are close in each layer. We show that such a multilayer neural field is more robust than conventional models when input patterns are deformed by noise disturbances. Moreover, its generalization ability can be slightly superior to that of conventional models.