AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens

TL;DR

AdaPerceiver introduces a unified transformer architecture capable of adaptively adjusting depth, width, and tokens, enabling efficient deployment across diverse tasks and hardware constraints while maintaining high performance.

Contribution

It is the first transformer model with integrated adaptivity across multiple axes, supported by a joint training regime for consistent performance.

Findings

Achieves higher accuracy-throughput trade-offs on image classification.

Reduces FLOPs significantly in dense prediction tasks while maintaining accuracy.

Maintains ImageNet accuracy with substantial FLOPs reduction.

Abstract

Modern transformer architectures achieve remarkable performance across tasks and domains but remain rigid in how they allocate computation at inference time. Real-world deployment often requires models to adapt to diverse hardware and latency constraints, yet most approaches to dynamic computation focus on a single axis -- such as reducing the number of tokens. We present a novel capability: AdaPerceiver, the first transformer architecture with unified adaptivity across depth, width, and tokens within a single model. We propose an architecture that supports adaptivity along these axes. We couple this with an efficient joint training regime that ensures the model maintains performance across its various configurations. We evaluate AdaPerceiver on image classification, semantic segmentation, and depth estimation tasks. On image classification, AdaPerceiver expands the accuracy-throughput Pareto front. It achieves 85.4% accuracy while yielding 36% higher throughput than FlexiViT-L. On dense prediction, AdaPerceiver matches ViT-H/14 while having $\sim$26x fewer encoder FLOPs (floating-point operations) on semantic segmentation and depth estimation. Finally, we show how AdaPerceiver equipped with a policy can maintain ImageNet1K accuracy ($\pm0.1$ percentage points) while reducing FLOPs by $24-33$%.