Reversible Vision Transformers

TL;DR

Reversible Vision Transformers significantly reduce memory usage in visual recognition tasks by enabling reversible architectures, allowing for larger models and faster training throughput without sacrificing accuracy.

Contribution

This work introduces reversible variants of Vision Transformers, enabling scalable, memory-efficient models suitable for resource-limited training environments.

Findings

Memory footprint reduced by up to 15.5x

Throughput increased by up to 2.3x for deeper models

Achieved comparable accuracy across multiple tasks

Abstract

We present Reversible Vision Transformers, a memory efficient architecture design for visual recognition. By decoupling the GPU memory requirement from the depth of the model, Reversible Vision Transformers enable scaling up architectures with efficient memory usage. We adapt two popular models, namely Vision Transformer and Multiscale Vision Transformers, to reversible variants and benchmark extensively across both model sizes and tasks of image classification, object detection and video classification. Reversible Vision Transformers achieve a reduced memory footprint of up to 15.5x at roughly identical model complexity, parameters and accuracy, demonstrating the promise of reversible vision transformers as an efficient backbone for hardware resource limited training regimes. Finally, we find that the additional computational burden of recomputing activations is more than overcome for deeper models, where throughput can increase up to 2.3x over their non-reversible counterparts. Full code and trained models are available at https://github.com/facebookresearch/slowfast. A simpler, easy to understand and modify version is also available at https://github.com/karttikeya/minREV