Slicing Vision Transformer for Flexible Inference

TL;DR

This paper introduces Scala, a framework that enables a single Vision Transformer to efficiently represent multiple sub-networks of varying widths, allowing flexible inference under resource constraints without retraining.

Contribution

Scala is a novel method that trains a ViT to support multiple sub-networks simultaneously, improving efficiency and performance over traditional separate training methods.

Findings

Scala matches the performance of separately trained ViTs.

Achieves 1.6% higher accuracy on ImageNet-1K.

Reduces parameters compared to prior methods.

Abstract

Vision Transformers (ViT) is known for its scalability. In this work, we target to scale down a ViT to fit in an environment with dynamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths. Thus, we propose a general framework, named Scala, to enable a single network to represent multiple smaller ViTs with flexible inference capability, which aligns with the inherent design of ViT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations on different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original ViT structure and matches the performance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% on ImageNet-1K with fewer parameters.