Universal Approximation Theorems of Fully Connected Binarized Neural Networks

TL;DR

This paper establishes universal approximation theorems for fully connected Binarized Neural Networks, demonstrating their capacity to approximate a wide class of functions under specific conditions, thus providing important theoretical guarantees.

Contribution

The paper provides the first comprehensive UA theorems for fully connected BNNs, covering both binarized and real-valued inputs, with constructive proofs for different network depths.

Findings

UA achievable with one hidden layer for binarized inputs

UA achievable with two hidden layers for Lipschitz functions with real inputs

Fully connected BNNs can universally approximate functions under certain conditions

Abstract

Neural networks (NNs) are known for their high predictive accuracy in complex learning problems. Beside practical advantages, NNs also indicate favourable theoretical properties such as universal approximation (UA) theorems. Binarized Neural Networks (BNNs) significantly reduce time and memory demands by restricting the weight and activation domains to two values. Despite the practical advantages, theoretical guarantees based on UA theorems of BNNs are rather sparse in the literature. We close this gap by providing UA theorems for fully connected BNNs under the following scenarios: (1) for binarized inputs, UA can be constructively achieved under one hidden layer; (2) for inputs with real numbers, UA can not be achieved under one hidden layer but can be constructively achieved under two hidden layers for Lipschitz-continuous functions. Our results indicate that fully connected BNNs can approximate functions universally, under certain conditions.