Are All Layers Created Equal?
Chiyuan Zhang, Samy Bengio, Yoram Singer

TL;DR
This paper investigates the functional heterogeneity of layers in overparameterized deep neural networks, revealing that some layers are robust to re-initialization while others are critical, impacting network performance.
Contribution
The study provides empirical evidence of layer heterogeneity, categorizing layers as robust or critical, and highlights the importance of architecture-aware robustness analysis.
Findings
Robust layers can be re-initialized without performance loss.
Critical layers' re-initialization significantly degrades accuracy.
Layer heterogeneity challenges simple parameter-based explanations.
Abstract
Understanding deep neural networks is a major research objective with notable experimental and theoretical attention in recent years. The practical success of excessively large networks underscores the need for better theoretical analyses and justifications. In this paper we focus on layer-wise functional structure and behavior in overparameterized deep models. To do so, we study empirically the layers' robustness to post-training re-initialization and re-randomization of the parameters. We provide experimental results which give evidence for the heterogeneity of layers. Morally, layers of large deep neural networks can be categorized as either "robust" or "critical". Resetting the robust layers to their initial values does not result in adverse decline in performance. In many cases, robust layers hardly change throughout training. In contrast, re-initializing critical layers vastly…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Stochastic Gradient Optimization Techniques · Machine Learning and Data Classification
