Expressive probabilistic sampling in recurrent neural networks

TL;DR

This paper introduces reservoir-sampler networks (RSNs), a novel recurrent neural architecture capable of sampling from complex distributions, advancing probabilistic brain models through efficient training and neural dynamics.

Contribution

It proposes RSNs with separate output units for complex distribution sampling and introduces a denoising score matching training method.

Findings

RSNs can sample from complex data distributions.

The proposed training method effectively implements Langevin sampling.

Traditional models have limited capacity for complex distribution sampling.

Abstract

In sampling-based Bayesian models of brain function, neural activities are assumed to be samples from probability distributions that the brain uses for probabilistic computation. However, a comprehensive understanding of how mechanistic models of neural dynamics can sample from arbitrary distributions is still lacking. We use tools from functional analysis and stochastic differential equations to explore the minimum architectural requirements for $\textit{recurrent}$ neural circuits to sample from complex distributions. We first consider the traditional sampling model consisting of a network of neurons whose outputs directly represent the samples (sampler-only network). We argue that synaptic current and firing-rate dynamics in the traditional model have limited capacity to sample from a complex probability distribution. We show that the firing rate dynamics of a recurrent neural circuit with a separate set of output units can sample from an arbitrary probability distribution. We call such circuits reservoir-sampler networks (RSNs). We propose an efficient training procedure based on denoising score matching that finds recurrent and output weights such that the RSN implements Langevin sampling. We empirically demonstrate our model's ability to sample from several complex data distributions using the proposed neural dynamics and discuss its applicability to developing the next generation of sampling-based brain models.