ConTNet: Why not use convolution and transformer at the same time?

TL;DR

ConTNet combines convolutional networks with transformers to capture global information efficiently, achieving high accuracy and robustness in image classification and detection tasks with less computational cost.

Contribution

This work introduces ConTNet, a novel architecture that integrates convolution and transformer components, offering improved performance and robustness over existing models like ViT and ResNet.

Findings

ConTNet achieves 81.8% top-1 accuracy on ImageNet.

ConTNet outperforms ResNet50 as a backbone in object detection tasks.

ConTNet has less than 40% of the computational complexity of DeiT-B.

Abstract

Although convolutional networks (ConvNets) have enjoyed great success in computer vision (CV), it suffers from capturing global information crucial to dense prediction tasks such as object detection and segmentation. In this work, we innovatively propose ConTNet (ConvolutionTransformer Network), combining transformer with ConvNet architectures to provide large receptive fields. Unlike the recently-proposed transformer-based models (e.g., ViT, DeiT) that are sensitive to hyper-parameters and extremely dependent on a pile of data augmentations when trained from scratch on a midsize dataset (e.g., ImageNet1k), ConTNet can be optimized like normal ConvNets (e.g., ResNet) and preserve an outstanding robustness. It is also worth pointing that, given identical strong data augmentations, the performance improvement of ConTNet is more remarkable than that of ResNet. We present its superiority and effectiveness on image classification and downstream tasks. For example, our ConTNet achieves 81.8% top-1 accuracy on ImageNet which is the same as DeiT-B with less than 40% computational complexity. ConTNet-M also outperforms ResNet50 as the backbone of both Faster-RCNN (by 2.6%) and Mask-RCNN (by 3.2%) on COCO2017 dataset. We hope that ConTNet could serve as a useful backbone for CV tasks and bring new ideas for model design