EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification

TL;DR

EVCC is a novel hybrid vision architecture that combines Transformers and CNNs with adaptive mechanisms, achieving state-of-the-art image classification accuracy while significantly reducing computational costs across multiple datasets.

Contribution

The paper introduces EVCC, a multi-branch architecture integrating Transformers and CNNs with adaptive token pruning, cross-attention, and dynamic routing for efficient and accurate image classification.

Findings

Outperforms state-of-the-art models like DeiT-Base and MaxViT-Base.

Achieves up to 2% higher accuracy on multiple datasets.

Reduces FLOPs by 25-35% compared to existing models.

Abstract

Hybrid vision architectures combining Transformers and CNNs have significantly advanced image classification, but they usually do so at significant computational cost. We introduce EVCC (Enhanced Vision Transformer-ConvNeXt-CoAtNet), a novel multi-branch architecture integrating the Vision Transformer, lightweight ConvNeXt, and CoAtNet through key innovations: (1) adaptive token pruning with information preservation, (2) gated bidirectional cross-attention for enhanced feature refinement, (3) auxiliary classification heads for multi-task learning, and (4) a dynamic router gate employing context-aware confidence-driven weighting. Experiments across the CIFAR-100, Tobacco3482, CelebA, and Brain Cancer datasets demonstrate EVCC's superiority over powerful models like DeiT-Base, MaxViT-Base, and CrossViT-Base by consistently achieving state-of-the-art accuracy with improvements of up to 2 percentage points, while reducing FLOPs by 25 to 35%. Our adaptive architecture adjusts computational demands to deployment needs by dynamically reducing token count, efficiently balancing the accuracy-efficiency trade-off while combining global context, local details, and hierarchical features for real-world applications. The source code of our implementation is available at https://anonymous.4open.science/r/EVCC.