Universal Approximation of Markov Kernels by Shallow Stochastic Feedforward Networks

TL;DR

This paper proves that shallow stochastic neural networks with sigmoid activations can universally approximate any Markov kernel, providing explicit bounds on the number of hidden units needed for such approximation.

Contribution

It establishes the minimal hidden units required for universal approximation of Markov kernels by shallow stochastic networks with sigmoid activation.

Findings

Explicit upper bounds on hidden units for universal approximation.

Approximation of any probabilistic assignment of output states given input states.

Provides theoretical foundation for the expressive power of shallow stochastic networks.

Abstract

We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of $n$ output units, given the states of $k\geq1$ input units, can be approximated arbitrarily well by a network with $2^{k-1}(2^{n-1}-1)$ hidden units.