A Lower Bound for the Number of Linear Regions of Ternary ReLU Regression Neural Networks

TL;DR

This paper provides a theoretical lower bound on the number of linear regions in ternary ReLU neural networks, explaining their expressivity and practical success.

Contribution

It establishes polynomial growth of linear regions with width and exponential with depth, and shows how to achieve bounds comparable to standard NNs.

Findings

Number of linear regions grows polynomially with width and exponentially with depth.

Doubling width and adjusting depth achieves bounds similar to standard ReLU networks.

Theoretical explanation for the practical success of ternary neural networks.

Abstract

With the advancement of deep learning, reducing computational complexity and memory consumption has become a critical challenge, and ternary neural networks (NNs) that restrict parameters to $\{-1, 0, +1\}$ have attracted attention as a promising approach. While ternary NNs demonstrate excellent performance in practical applications such as image recognition and natural language processing, their theoretical understanding remains insufficient. In this paper, we theoretically analyze the expressivity of ternary NNs from the perspective of the number of linear regions. Specifically, we evaluate the number of linear regions of ternary regression NNs with Rectified Linear Unit (ReLU) for activation functions and prove that the number of linear regions increases polynomially with respect to network width and exponentially with respect to depth, similar to standard NNs. Moreover, we show that it suffices to first double the width, then either square the width or double the depth of ternary NNs with alternating ReLU and identity layers to achieve a lower bound on the maximum number of linear regions comparable to that of general ReLU regression NNs. When using ReLU in all the layers, a similar bound is obtained by further doubling the width. This provides a theoretical explanation, in some sense, for the practical success of ternary NNs.