CoCAViT: Compact Vision Transformer with Robust Global Coordination

TL;DR

CoCAViT introduces a robust, efficient vision transformer architecture that enhances out-of-distribution generalization and local-global feature modeling, achieving high accuracy and low latency on multiple benchmarks.

Contribution

The paper proposes CoCAViT, a novel vision transformer with a Coordinator-patch Cross Attention mechanism that improves robustness and performance of small models across diverse tasks.

Findings

Achieves 84.0% top-1 accuracy on ImageNet-1K with 28M parameters.

Significantly improves out-of-distribution benchmark performance.

Attains high accuracy on COCO detection and ADE20K segmentation with low latency.

Abstract

In recent years, large-scale visual backbones have demonstrated remarkable capabilities in learning general-purpose features from images via extensive pre-training. Concurrently, many efficient architectures have emerged that have performance comparable to that of larger models on in-domain benchmarks. However, we observe that for smaller models, the performance drop on out-of-distribution (OOD) data is disproportionately larger, indicating a deficiency in the generalization performance of existing efficient models. To address this, we identify key architectural bottlenecks and inappropriate design choices that contribute to this issue, retaining robustness for smaller models. To restore the global field of pure window attention, we further introduce a Coordinator-patch Cross Attention (CoCA) mechanism, featuring dynamic, domain-aware global tokens that enhance local-global feature modeling and adaptively capture robust patterns across domains with minimal computational overhead. Integrating these advancements, we present CoCAViT, a novel visual backbone designed for robust real-time visual representation. Extensive experiments empirically validate our design. At a resolution of 224*224, CoCAViT-28M achieves 84.0% top-1 accuracy on ImageNet-1K, with significant gains on multiple OOD benchmarks, compared to competing models. It also attains 52.2 mAP on COCO object detection and 51.3 mIOU on ADE20K semantic segmentation, while maintaining low latency.