TL;DR
This paper links the initial bias in deep neural networks to their trainability, revealing that optimal initialization involves a systematic bias rather than neutrality, which challenges conventional assumptions.
Contribution
It provides a theoretical proof connecting initial guessing bias to mean field theories, showing that trainability is influenced by inherent architectural biases.
Findings
Initial biases are linked to mean field properties of DNNs.
Optimal trainability correlates with systematic biases at initialization.
Unbiased initializations are not necessarily optimal for training.
Abstract
The statistical properties of deep neural networks (DNNs) at initialization play an important role to comprehend their trainability and the intrinsic architectural biases they possess before data exposure Well established mean field (MF) theories have uncovered that the distribution of parameters of randomly initialized networks strongly influences the behavior of the gradients, dictating whether they explode or vanish. Recent work has showed that untrained DNNs also manifest an initial guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof that links IGB to previous MF theories for a vast class of DNNs, showing that efficient learning is tightly connected to a network prejudice towards a specific class. This connection leads to a counterintuitive conclusion: the initialization that optimizes…
Peer Reviews
Decision·ICLR 2026 Poster
Understanding how and why neural networks are biased is an important topic for fairness in AI systems. This paper considers the impact of weight initialization, independent of data, which could help us understand how learning dynamics affect bias. By connecting IGB to established mean-field theory of wide networks, it helps build a framework for this. The authors support their theoretical claims with experiments in several architectures and datasets.
I found the paper difficult to follow and understand. I think it could have more strongly motivated why the link to MF theory is an important one to make and why the contribution made is important. It was also unclear to me at times what was background on previous work versus a novel contribution of the paper. “Prejudice” is a more loaded term than “bias” and I find the use of it here somewhat inappropriate for what is being referred to. Bias is used to refer to behavior a network might be more
1. Clean conceptual claim that ties IGB statistics with network trainability 2. Clear testable claim forhow models which start with maximal bias learn fastest 3. Robust checks beyond toy MLPs to ViTs, both training and perturbing them
1. The infinite width setting is somewhat limiting on the theory side 2. It's unclear how much the choice of norms matters or whether this result is idiosyncratic 3. The pooling for the 2-dimensional case is similarly limited
1. Theoretical contribution: The paper establishes a clean mathematical connection between two previously distinct frameworks (MF and IGB), enriching both perspectives. 2. Insight: It introduces a novel idea: bias at initialization can improve trainability; The new “prejudice-neutrality” phase view offers an intuitive explanation for initialization effects, linking bias to the dynamical stability of gradient flow.
1. The theory is derived in the infinite-width limit and validated on small- to mid-scale settings. Its applicability to practical, large-scale deep networks (e.g., transformers with normalization and attention) is not demonstrated. Additionally, empirical evaluation focuses on synthetic and small vision datasets. It is of readers' interest to learn results on simple language tasks. 2. Ambiguous practical relevance: While the “transient bias” insight is conceptually interesting, there is no cle
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
