A Graph Sufficiency Perspective for Neural Networks

TL;DR

This paper introduces a graph-based framework to analyze neural networks, establishing conditions under which layer outputs retain all necessary information for the target, thus providing a new statistical perspective on deep learning models.

Contribution

It develops a theoretical framework linking graph sufficiency to neural network layers, covering various architectures and providing bounds on loss and sufficiency conditions.

Findings

Asymptotic sufficiency in infinite-width networks

Exact or approximate sufficiency in finite-width networks

Framework applies to CNNs, ReLU, sigmoid, and fully connected layers

Abstract

This paper analyzes neural networks through graph variables and statistical sufficiency. We interpret neural network layers as graph-based transformations, where neurons act as pairwise functions between inputs and learned anchor points. Within this formulation, we establish conditions under which layer outputs are sufficient for the layer inputs, that is, each layer preserves the conditional distribution of the target variable given the input variable. We explore two theoretical paths under this graph-based view. The first path assumes dense anchor points and shows that asymptotic sufficiency holds in the infinite-width limit and is preserved throughout training. The second path, more aligned with practical architectures, proves exact or approximate sufficiency in finite-width networks by assuming region-separated input distributions and constructing appropriate anchor points. This path can ensure the sufficiency property for an infinite number of layers, and provide error bounds on the optimal loss for both regression and classification tasks using standard neural networks. Our framework covers fully connected layers, general pairwise functions, ReLU and sigmoid activations, and convolutional neural networks. Overall, this work bridges statistical sufficiency, graph-theoretic representations, and deep learning, providing a new statistical understanding of neural networks.