Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs

TL;DR

This paper introduces Mix-Shift-MLP, a simple yet effective model that exploits both global and local dependencies in vision tasks by increasing local receptive fields through shifting, achieving competitive results without self-attention.

Contribution

The paper proposes Mix-Shift-MLP, a novel architecture that combines mixing and shifting techniques to capture both global and local dependencies without using self-attention.

Findings

Achieves 83.8% top-1 accuracy on ImageNet-1K with 85M parameters.

Improves performance when combined with Vision Transformers like Swin Transformer.

Simple implementation with competitive benchmark results.

Abstract

Token-mixing multi-layer perceptron (MLP) models have shown competitive performance in computer vision tasks with a simple architecture and relatively small computational cost. Their success in maintaining computation efficiency is mainly attributed to avoiding the use of self-attention that is often computationally heavy, yet this is at the expense of not being able to mix tokens both globally and locally. In this paper, to exploit both global and local dependencies without self-attention, we present Mix-Shift-MLP (MS-MLP) which makes the size of the local receptive field used for mixing increase with respect to the amount of spatial shifting. In addition to conventional mixing and shifting techniques, MS-MLP mixes both neighboring and distant tokens from fine- to coarse-grained levels and then gathers them via a shifting operation. This directly contributes to the interactions between global and local tokens. Being simple to implement, MS-MLP achieves competitive performance in multiple vision benchmarks. For example, an MS-MLP with 85 million parameters achieves 83.8% top-1 classification accuracy on ImageNet-1K. Moreover, by combining MS-MLP with state-of-the-art Vision Transformers such as the Swin Transformer, we show MS-MLP achieves further improvements on three different model scales, e.g., by 0.5% on ImageNet-1K classification with Swin-B. The code is available at: https://github.com/JegZheng/MS-MLP.