Memory Efficient Transformer Adapter for Dense Predictions

TL;DR

META is a memory-efficient Vision Transformer adapter that reduces memory access operations, improves inference speed, and enhances dense prediction tasks by sharing normalization, employing cross-shaped self-attention, and adding lightweight convolutional features.

Contribution

The paper introduces META, a novel ViT adapter that significantly improves memory efficiency and inference speed while boosting accuracy for dense prediction tasks.

Findings

Achieves state-of-the-art accuracy-efficiency trade-off.

Reduces memory consumption and inference time.

Enhances dense prediction performance across multiple datasets.

Abstract

While current Vision Transformer (ViT) adapter methods have shown promising accuracy, their inference speed is implicitly hindered by inefficient memory access operations, e.g., standard normalization and frequent reshaping. In this work, we propose META, a simple and fast ViT adapter that can improve the model's memory efficiency and decrease memory time consumption by reducing the inefficient memory access operations. Our method features a memory-efficient adapter block that enables the common sharing of layer normalization between the self-attention and feed-forward network layers, thereby reducing the model's reliance on normalization operations. Within the proposed block, the cross-shaped self-attention is employed to reduce the model's frequent reshaping operations. Moreover, we augment the adapter block with a lightweight convolutional branch that can enhance local inductive biases, particularly beneficial for the dense prediction tasks, e.g., object detection, instance segmentation, and semantic segmentation. The adapter block is finally formulated in a cascaded manner to compute diverse head features, thereby enriching the variety of feature representations. Empirically, extensive evaluations on multiple representative datasets validate that META substantially enhances the predicted quality, while achieving a new state-of-the-art accuracy-efficiency trade-off. Theoretically, we demonstrate that META exhibits superior generalization capability and stronger adaptability.