Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stability

TL;DR

This paper investigates how sparse connectivity in convolutional networks influences generalization, revealing that small receptive fields and structured patch geometry enable better generalization below the Edge of Stability.

Contribution

It demonstrates that sparse connectivity alters the stability constraints, leading to non-vacuous generalization bounds, and explains the generalization advantage of convolutional architectures.

Findings

Sparse connectivity processes low-dimensional patches, improving generalization bounds.

Structured patch geometry in natural images facilitates better generalization.

Unstructured patches fail to prevent overfitting, highlighting the importance of data geometry.

Abstract

Gradient descent on overparameterized neural networks typically operates at the Edge of Stability (EoS), where the largest Hessian eigenvalue hovers around a step-size-dependent threshold. We study how sparse connectivity changes generalization below this threshold in two-layer ReLU networks. Prior results have shown that for fully-connected networks (FCNs), generalization guarantees in this regime degrade and become vacuous on high-dimensional spherical inputs. Our analysis reveals that sparse connectivity fundamentally alters this picture. Under sparse connectivity, the network processes a collection of low-dimensional patches rather than the full input vector, so the effective constraint imposed by the stability condition is governed by the geometry of the training patch collection. We prove that when the receptive fields are small relative to the ambient dimension, the effective constraint yields non-vacuous generalization bounds in precisely the spherical regime where FCNs provably fail. The same framework also reveals a contrasting failure mode: if the patch collection lacks geometric structure, the constraint becomes unable to prevent overfitting. We corroborate this theory by analyzing the patch geometry of natural images, showing that standard convolutional designs produce patch multiset with low-dimensional structure that facilitates generalization. This provides a principled explanation for the generalization advantage of convolutional networks. Thus, our analysis yields a unified framework that identifies how architecture, data geometry, and gradient descent jointly govern generalization performance.