Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

TL;DR

Vision Permutator introduces a simple, data-efficient MLP-like architecture that encodes positional information along height and width separately, achieving competitive accuracy on ImageNet without complex mechanisms.

Contribution

It proposes a novel way to encode spatial information in MLP-like models by separately permuting height and width, improving efficiency and accuracy.

Findings

Achieves 81.5% top-1 accuracy on ImageNet with 25M parameters.

Outperforms many CNNs and vision transformers of similar size.

Scales to 88M parameters with 83.2% accuracy.

Abstract

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies along one spatial direction and meanwhile preserve precise positional information along the other direction. The resulting position-sensitive outputs are then aggregated in a mutually complementing manner to form expressive representations of the objects of interest. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. Code is available at https://github.com/Andrew-Qibin/VisionPermutator.