Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

TL;DR

This paper demonstrates that integrating convolutional embedding layers into hierarchical Vision Transformers significantly enhances their performance across multiple vision tasks by providing beneficial inductive biases.

Contribution

The study systematically explores how macro architecture and convolutional embedding improve ViT performance, introducing CETNets as efficient hybrid CNN/ViT backbones.

Findings

CETNets achieve 84.9% Top-1 accuracy on ImageNet-1K

Effective boosting of state-of-the-art ViTs with convolutional embedding

Improved performance on COCO and ADE20K benchmarks

Abstract

Vision Transformers (ViTs) have recently dominated a range of computer vision tasks, yet it suffers from low training data efficiency and inferior local semantic representation capability without appropriate inductive bias. Convolutional neural networks (CNNs) inherently capture regional-aware semantics, inspiring researchers to introduce CNNs back into the architecture of the ViTs to provide desirable inductive bias for ViTs. However, is the locality achieved by the micro-level CNNs embedded in ViTs good enough? In this paper, we investigate the problem by profoundly exploring how the macro architecture of the hybrid CNNs/ViTs enhances the performances of hierarchical ViTs. Particularly, we study the role of token embedding layers, alias convolutional embedding (CE), and systemically reveal how CE injects desirable inductive bias in ViTs. Besides, we apply the optimal CE configuration to 4 recently released state-of-the-art ViTs, effectively boosting the corresponding performances. Finally, a family of efficient hybrid CNNs/ViTs, dubbed CETNets, are released, which may serve as generic vision backbones. Specifically, CETNets achieve 84.9% Top-1 accuracy on ImageNet-1K (training from scratch), 48.6% box mAP on the COCO benchmark, and 51.6% mIoU on the ADE20K, substantially improving the performances of the corresponding state-of-the-art baselines.