AdaptViG: Adaptive Vision GNN with Exponential Decay Gating

TL;DR

AdaptViG introduces an efficient hybrid Vision GNN with adaptive graph construction and exponential decay gating, achieving state-of-the-art accuracy-efficiency trade-offs in vision tasks.

Contribution

It proposes a novel adaptive graph construction mechanism with exponential decay gating and a hybrid strategy for improved efficiency and accuracy in Vision GNNs.

Findings

Achieves 82.6% top-1 accuracy with fewer parameters and GMACs.

Outperforms larger models on downstream tasks.

Provides a new state-of-the-art efficiency-accuracy balance.

Abstract

Vision Graph Neural Networks (ViGs) offer a new direction for advancements in vision architectures. While powerful, ViGs often face substantial computational challenges stemming from their graph construction phase, which can hinder their efficiency. To address this issue we propose AdaptViG, an efficient and powerful hybrid Vision GNN that introduces a novel graph construction mechanism called Adaptive Graph Convolution. This mechanism builds upon a highly efficient static axial scaffold and a dynamic, content-aware gating strategy called Exponential Decay Gating. This gating mechanism selectively weighs long-range connections based on feature similarity. Furthermore, AdaptViG employs a hybrid strategy, utilizing our efficient gating mechanism in the early stages and a full Global Attention block in the final stage for maximum feature aggregation. Our method achieves a new state-of-the-art trade-off between accuracy and efficiency among Vision GNNs. For instance, our AdaptViG-M achieves 82.6% top-1 accuracy, outperforming ViG-B by 0.3% while using 80% fewer parameters and 84% fewer GMACs. On downstream tasks, AdaptViG-M obtains 45.8 mIoU, 44.8 APbox, and 41.1 APmask, surpassing the much larger EfficientFormer-L7 by 0.7 mIoU, 2.2 APbox, and 2.1 APmask, respectively, with 78% fewer parameters.