Sliced Recursive Transformer

TL;DR

The paper introduces Sliced Recursive Transformer (SReT), a parameter-efficient vision transformer that shares weights across layers, improves accuracy, reduces computational costs, and enables scalable deep models with minimal overhead.

Contribution

It proposes a novel weight sharing recursive structure with sliced group self-attentions, enhancing efficiency and scalability of vision transformers without extra parameters.

Findings

Achieves ~2% accuracy gain on ImageNet-1K with recursive weight sharing.

Reduces computational cost by 10-30% using sliced group self-attentions.

Enables construction of very deep transformers with over 100 shared layers.

Abstract

We present a neat yet effective recursive operation on vision transformers that can improve parameter utilization without involving additional parameters. This is achieved by sharing weights across the depth of transformer networks. The proposed method can obtain a substantial gain (~2%) simply using naive recursive operation, requires no special or sophisticated knowledge for designing principles of networks, and introduces minimal computational overhead to the training procedure. To reduce the additional computation caused by recursive operation while maintaining the superior accuracy, we propose an approximating method through multiple sliced group self-attentions across recursive layers which can reduce the cost consumption by 10~30% with minimal performance loss. We call our model Sliced Recursive Transformer (SReT), a novel and parameter-efficient vision transformer design that is compatible with a broad range of other designs for efficient ViT architectures. Our best model establishes significant improvement on ImageNet-1K over state-of-the-art methods while containing fewer parameters. The proposed weight sharing mechanism by sliced recursion structure allows us to build a transformer with more than 100 or even 1000 shared layers with ease while keeping a compact size (13~15M), to avoid optimization difficulties when the model is too large. The flexible scalability has shown great potential for scaling up models and constructing extremely deep vision transformers. Code is available at https://github.com/szq0214/SReT.