GNN-MoE: Context-Aware Patch Routing using GNNs for Parameter-Efficient Domain Generalization

TL;DR

GNN-MoE introduces a graph neural network-based routing mechanism for parameter-efficient domain generalization in vision transformers, leveraging inter-patch relationships for improved robustness and efficiency.

Contribution

It proposes a novel GNN-based patch routing method within a Mixture-of-Experts framework to enhance domain generalization while maintaining parameter efficiency.

Findings

Achieves state-of-the-art or competitive results on DG benchmarks.

Demonstrates high parameter efficiency with GNN-based routing.

Leverages inter-patch relationships for better domain adaptation.

Abstract

Domain generalization (DG) seeks robust Vision Transformer (ViT) performance on unseen domains. Efficiently adapting pretrained ViTs for DG is challenging; standard fine-tuning is costly and can impair generalization. We propose GNN-MoE, enhancing Parameter-Efficient Fine-Tuning (PEFT) for DG with a Mixture-of-Experts (MoE) framework using efficient Kronecker adapters. Instead of token-based routing, a novel Graph Neural Network (GNN) router (GCN, GAT, SAGE) operates on inter-patch graphs to dynamically assign patches to specialized experts. This context-aware GNN routing leverages inter-patch relationships for better adaptation to domain shifts. GNN-MoE achieves state-of-the-art or competitive DG benchmark performance with high parameter efficiency, highlighting the utility of graph-based contextual routing for robust, lightweight DG.