Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

TL;DR

This paper demonstrates that the spectral bifurcation of the Hessian in deep neural networks can originate solely from network architecture, independent of data imbalance, with spectral gap scaling linearly with depth.

Contribution

It provides a theoretical analysis showing that spectral bifurcation arises from architecture alone, challenging previous data-centric explanations.

Findings

Hessian spectral bifurcation can occur without data imbalance.

The spectral gap scales linearly with network depth.

Architecture influences the Hessian eigenvalue distribution significantly.

Abstract

The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.