Additive function approximation in the brain

TL;DR

This paper explores how sparse neural networks in the brain can be understood as additive models, revealing their advantages in function approximation, stability, and scalability, and bridging neuroscience with kernel theory.

Contribution

It characterizes the function spaces induced by sparse random features and highlights the benefits of sparsity in biological neural networks.

Findings

Sparse networks act as additive models of various orders.

Sparsity reduces the curse of dimensionality in learning.

Sparse features are scalable and robust to noise.

Abstract

Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons. For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned. A network with $d$ inputs per neuron is found to be equivalent to an additive model of order $d$, whereas with a degree distribution the network combines additive terms of different orders. We identify three specific advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable. Thus, even simple brain architectures can be powerful function approximators. Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists.