Almost Uniform Sampling From Neural Networks

TL;DR

This paper presents algorithms for approximately uniform sampling of neural network labelings, with guarantees on probability and efficiency, including an exact sampling method for single neurons.

Contribution

It introduces a polynomial-time algorithm for nearly uniform sampling of labelings in neural networks with fixed architecture and provides an exact sampling method for single neurons.

Findings

Algorithm runs in polynomial time in sample size and network weights.

Probability of any labeling appearing is at least (W/2ekn)^W.

Exact uniform sampling achieved for single neuron case.

Abstract

Given a length $n$ sample from $\mathbb{R}^d$ and a neural network with a fixed architecture with $W$ weights, $k$ neurons, linear threshold activation functions, and binary outputs on each neuron, we study the problem of uniformly sampling from all possible labelings on the sample corresponding to different choices of weights. We provide an algorithm that runs in time polynomial both in $n$ and $W$ such that any labeling appears with probability at least $\left(\frac{W}{2ekn}\right)^W$ for $W<n$. For a single neuron, we also provide a random walk based algorithm that samples exactly uniformly.