SPIN: An Empirical Evaluation on Sharing Parameters of Isotropic Networks

TL;DR

This paper empirically evaluates parameter sharing strategies in isotropic neural networks like ConvMixer and vision transformers, proposing a new method that improves efficiency and accuracy over traditional scaling.

Contribution

It introduces a formal framework for weight sharing in isotropic networks and proposes a strategy that enhances model efficiency and performance.

Findings

Achieved 1.9x compression of ConvMixer with improved accuracy on ImageNet.

Provided a comprehensive empirical evaluation of weight sharing design choices.

Developed a qualitative understanding of weight sharing behavior in isotropic architectures.

Abstract

Recent isotropic networks, such as ConvMixer and vision transformers, have found significant success across visual recognition tasks, matching or outperforming non-isotropic convolutional neural networks (CNNs). Isotropic architectures are particularly well-suited to cross-layer weight sharing, an effective neural network compression technique. In this paper, we perform an empirical evaluation on methods for sharing parameters in isotropic networks (SPIN). We present a framework to formalize major weight sharing design decisions and perform a comprehensive empirical evaluation of this design space. Guided by our experimental results, we propose a weight sharing strategy to generate a family of models with better overall efficiency, in terms of FLOPs and parameters versus accuracy, compared to traditional scaling methods alone, for example compressing ConvMixer by 1.9x while improving accuracy on ImageNet. Finally, we perform a qualitative study to further understand the behavior of weight sharing in isotropic architectures. The code is available at https://github.com/apple/ml-spin.