When Does Sparsity Mitigate the Curse of Depth in LLMs

TL;DR

This paper shows that sparsity, both implicit and explicit, helps mitigate the curse of depth in large language models by regulating variance propagation, leading to better layer utilization and improved downstream task performance.

Contribution

The study reveals sparsity as a key mechanism for enhancing depth utilization in LLMs, supported by experiments and practical guidelines, and introduces a new understanding of sparsity's role beyond efficiency.

Findings

Sparsity reduces output variance and enhances layer differentiation.

Sparsity improves downstream task accuracy by 4.6%.

Implicit and explicit sparsity both contribute to better depth scaling.

Abstract

Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we demonstrate that, sparsity, beyond enabling efficiency, acts as a regulator of variance propagation and thereby improves depth utilization. Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing sparsity in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: sparsity improves layer utilization by reducing output variance and promoting functional differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training deptheffective LLMs, yielding a notable 4.6% accuracy improvement on downstream tasks. Our results reveal sparsity, arising naturally from standard design choices, as a key yet previously overlooked mechanism for effective depth scaling in LLMs. Code is available at https://github.com/pUmpKin-Co/SparsityAndCoD.