ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters

TL;DR

ScaleNet provides a cost-effective method for expanding pretrained vision transformer models by adding layers with shared weights and minimal additional parameters, leading to improved performance and efficiency.

Contribution

We introduce ScaleNet, a novel approach for efficiently scaling pretrained ViT models through layer insertion and weight sharing, reducing training costs while maintaining or improving accuracy.

Findings

Achieves 7.42% accuracy improvement on ImageNet-1K with fewer epochs.

Enables efficient model expansion with negligible parameter increase.

Shows potential for downstream vision tasks like object detection.

Abstract

Recent advancements in vision transformers (ViTs) have demonstrated that larger models often achieve superior performance. However, training these models remains computationally intensive and costly. To address this challenge, we introduce ScaleNet, an efficient approach for scaling ViT models. Unlike conventional training from scratch, ScaleNet facilitates rapid model expansion with negligible increases in parameters, building on existing pretrained models. This offers a cost-effective solution for scaling up ViTs. Specifically, ScaleNet achieves model expansion by inserting additional layers into pretrained ViTs, utilizing layer-wise weight sharing to maintain parameters efficiency. Each added layer shares its parameter tensor with a corresponding layer from the pretrained model. To mitigate potential performance degradation due to shared weights, ScaleNet introduces a small set of adjustment parameters for each layer. These adjustment parameters are implemented through parallel adapter modules, ensuring that each instance of the shared parameter tensor remains distinct and optimized for its specific function. Experiments on the ImageNet-1K dataset demonstrate that ScaleNet enables efficient expansion of ViT models. With a 2$\times$ depth-scaled DeiT-Base model, ScaleNet achieves a 7.42% accuracy improvement over training from scratch while requiring only one-third of the training epochs, highlighting its efficiency in scaling ViTs. Beyond image classification, our method shows significant potential for application in downstream vision areas, as evidenced by the validation in object detection task.