The Local Learning Coefficient: A Singularity-Aware Complexity Measure

TL;DR

The paper introduces the Local Learning Coefficient (LLC), a new complexity measure for deep neural networks that accounts for singularities in the loss landscape, providing insights into model complexity and training heuristics.

Contribution

It defines and explores the LLC based on Singular Learning Theory, proposes a scalable estimator, and demonstrates its application across various neural network architectures.

Findings

LLC offers insights into DNN complexity and training effects.

The estimator scales to large models like ResNets and transformers.

LLC helps reconcile deep learning complexity with parsimony principles.

Abstract

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC's theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony.