Critical dynamics governs deep learning

TL;DR

This paper proposes that the criticality of network dynamics is fundamental to deep learning performance, robustness, and resilience, and demonstrates how maintaining criticality improves AI models and explains their failures.

Contribution

It introduces the concept that criticality governs DNN dynamics, shows that successful models tend toward criticality, and develops methods to maintain criticality for improved AI robustness.

Findings

Successful models tend toward criticality over time.

Explicitly maintaining criticality improves robustness and accuracy.

Loss of criticality explains AI pathologies like collapse and degradation.

Abstract

The rapid advances in artificial intelligence (AI) have largely been driven by scaling deep neural networks (DNNs) - increasing model size, data, and computational resources. Yet performance is ultimately governed by network dynamics. The lack of a principled understanding of DNN dynamics beyond heuristic design has contributed to challenges in robustness, suboptimal performance, high energy consumption, and pathologies in continual and AI-generated content learning. In contrast, the human brain appears largely resilient to these problems, and converging evidence suggests this advantage arises from dynamics poised at a critical phase transition. Inspired by this principle, we propose that criticality provides a unifying framework linking structure, dynamics, and function in DNNs. First, analyzing more than 80 state-of-the-art models, we show that a decade of AI progress has implicitly driven successful networks toward criticality - explaining why some architectures succeeded while others failed. Second, we demonstrate that explicitly incorporating criticality into training improves robustness and accuracy while mitigating key limitations of current models. Third, we show that major AI pathologies - including performance degradation in continual learning and model collapse during training on AI-generated data - reflect a loss of critical dynamics. By maintaining networks near criticality, we provide a principled solution to these failures, demonstrating that criticality-based optimization prevents degradation and collapse. Our results establish criticality as a substrate-independent principle of intelligence, connecting AI progress with fundamental principles of brain function, and offering both theoretical insight and practical strategies to ensure long-term DNN performance and resilience as models scale.