On Emergences of Non-Classical Statistical Characteristics in Classical Neural Networks

TL;DR

This paper introduces NCnet, a classical neural network exhibiting non-classical statistical behaviors inspired by quantum mechanics, revealing insights into internal interactions and training dynamics.

Contribution

The paper proposes NCnet, a simple classical neural architecture that demonstrates non-classical statistical behaviors and correlations without physical communication links.

Findings

Non-classicality measured by CHSH inequality arises from gradient competitions.

Implicit sensing of task training occurs via local loss oscillations.

Non-classical statistics correlate with generalization performance.

Abstract

Inspired by measurement incompatibility and Bell-family inequalities in quantum mechanics, we propose the Non-Classical Network (NCnet), a simple classical neural architecture that stably exhibits non-classical statistical behaviors under typical and interpretable experimental setups. We find non-classicality, measured by the $S$ statistic of CHSH inequality, arises from gradient competitions of hidden-layer neurons shared by multi-tasks. Remarkably, even without physical links supporting explicit communication, one task head can implicitly sense the training task of other task heads via local loss oscillations, leading to non-local correlations in their training outcomes. Specifically, in the low-resource regime, the value of $S$ increases gradually with increasing resources and approaches toward its classical upper-bound 2, which implies that underfitting is alleviated with resources increase. As the model nears the critical scale required for adequate performance, $S$ may temporarily exceed 2. As resources continue to grow, $S$ then asymptotically decays down to and fluctuates around 2. Empirically, when model capacity is insufficient, $S$ is positively correlated with generalization performance, and the regime where $S$ first approaches $2$ often corresponding to good generalization. Overall, our results suggest that non-classical statistics can provide a novel perspective for understanding internal interactions and training dynamics of deep networks.