Interpret Vision Transformers as ConvNets with Dynamic Convolutions

TL;DR

This paper unifies vision Transformers and ConvNets under a common framework of dynamic convolutions, enabling direct comparison and guiding new network designs that improve efficiency and performance.

Contribution

It introduces a unified interpretation of vision Transformers as ConvNets with dynamic convolutions, facilitating design insights and novel architectures.

Findings

Replacing softmax with ConvNet modules improves convergence and performance.

Designing depth-wise vision Transformers yields more efficient models with comparable accuracy.

Abstract

There has been a debate about the superiority between vision Transformers and ConvNets, serving as the backbone of computer vision models. Although they are usually considered as two completely different architectures, in this paper, we interpret vision Transformers as ConvNets with dynamic convolutions, which enables us to characterize existing Transformers and dynamic ConvNets in a unified framework and compare their design choices side by side. In addition, our interpretation can also guide the network design as researchers now can consider vision Transformers from the design space of ConvNets and vice versa. We demonstrate such potential through two specific studies. First, we inspect the role of softmax in vision Transformers as the activation function and find it can be replaced by commonly used ConvNets modules, such as ReLU and Layer Normalization, which results in a faster convergence rate and better performance. Second, following the design of depth-wise convolution, we create a corresponding depth-wise vision Transformer that is more efficient with comparable performance. The potential of the proposed unified interpretation is not limited to the given examples and we hope it can inspire the community and give rise to more advanced network architectures.