On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

TL;DR

This paper investigates the expressive power of quantized neural networks with fixed-point parameters and inexact operations, establishing conditions under which they can represent all fixed-point functions, including networks with binary weights.

Contribution

It provides necessary and sufficient conditions for quantized networks to represent all fixed-point functions, covering various activation functions and binary weight networks.

Findings

Popular activation functions satisfy the sufficient condition.

Quantized networks can represent all fixed-point functions under certain conditions.

Binary weight networks can also represent all fixed-point functions.

Abstract

Existing works on the expressive power of neural networks typically assume real parameters and exact operations. In this work, we study the expressive power of quantized networks under discrete fixed-point parameters and inexact fixed-point operations with round-off errors. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for quantized networks to represent all fixed-point functions from fixed-point vectors to fixed-point numbers. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of representing all fixed-point functions. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function $\sigma$, there exists a fixed-point number $x$ such that $\sigma(x)=0$. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in $\{-1,1\}$ can also represent all fixed-point functions for practical activation functions.