The Expressive Power of Tuning Only the Normalization Layers

TL;DR

This paper investigates the expressive power of tuning only normalization layers in deep neural networks, showing that such tuning can reconstruct large classes of target networks, highlighting their significant role.

Contribution

It demonstrates that tuning only normalization layers in random ReLU networks can reconstruct networks up to a certain size, revealing their substantial expressive capacity.

Findings

Tuning normalization layers can reconstruct target networks up to a size proportional to the square root of network width.

This holds even for sparsified networks with sufficient overparameterization.

Normalization tuning can achieve high accuracy in downstream tasks with minimal parameter updates.

Abstract

Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks. Recent studies on fine-tuning large pretrained models indicate that just tuning the parameters of these affine transforms can achieve high accuracy for downstream tasks. These findings open the questions about the expressive power of tuning the normalization layers of frozen networks. In this work, we take the first step towards this question and show that for random ReLU networks, fine-tuning only its normalization layers can reconstruct any target network that is $O(\sqrt{\text{width}})$ times smaller. We show that this holds even for randomly sparsified networks, under sufficient overparameterization, in agreement with prior empirical work.