Expressive power of binary and ternary neural networks

TL;DR

This paper investigates the approximation capabilities of deep sparse ReLU networks with binary and ternary weights, demonstrating their ability to approximate certain classes of functions on high-dimensional spaces.

Contribution

It establishes the expressive power of binary and ternary neural networks in approximating Hölder and continuous functions, highlighting their theoretical capabilities.

Findings

Binary and ternary networks can approximate Hölder functions.

Depth-2 binary networks with indicator activation functions can approximate continuous functions.

Theoretical bounds on approximation capabilities of low-precision neural networks.

Abstract

We show that deep sparse ReLU networks with ternary weights and deep ReLU networks with binary weights can approximate $\beta$-H\"older functions on $[0,1]^d$. Also, for any interval $[a,b)\subset\mathbb{R}$, continuous functions on $[0,1]^d$ can be approximated by networks of depth $2$ with binary activation function $\mathds{1}_{[a,b)}$.