Learning Boolean Circuits with Neural Networks

TL;DR

This paper investigates how neural networks learn Boolean circuits, revealing that local correlations influence learning success, and demonstrates depth separation where deep networks outperform shallow ones in expressing certain functions.

Contribution

The paper introduces the role of local correlation in neural network learning of Boolean circuits and establishes a depth separation result using communication complexity.

Findings

Neural networks can learn the (log n)-parity problem under most product distributions.

Local correlation between circuit gates and labels determines optimization success.

Shallow networks cannot express certain functions that deep networks can learn efficiently.

Abstract

While on some natural distributions, neural-networks are trained efficiently using gradient-based algorithms, it is known that learning them is computationally hard in the worst-case. To separate hard from easy to learn distributions, we observe the property of local correlation: correlation between local patterns of the input and the target label. We focus on learning deep neural-networks using a gradient-based algorithm, when the target function is a tree-structured Boolean circuit. We show that in this case, the existence of correlation between the gates of the circuit and the target label determines whether the optimization succeeds or fails. Using this result, we show that neural-networks can learn the (log n)-parity problem for most product distributions. These results hint that local correlation may play an important role in separating easy/hard to learn distributions. We also obtain a novel depth separation result, in which we show that a shallow network cannot express some functions, while there exists an efficient gradient-based algorithm that can learn the very same functions using a deep network. The negative expressivity result for shallow networks is obtained by a reduction from results in communication complexity, that may be of independent interest.