PCaM: A Progressive Focus Attention-Based Information Fusion Method for Improving Vision Transformer Domain Adaptation

TL;DR

This paper introduces PCaM, a novel attention mechanism for Vision Transformers that progressively filters background information to enhance foreground focus, significantly improving unsupervised domain adaptation performance across multiple datasets.

Contribution

The paper proposes a lightweight, architecture-agnostic progressive focus cross-attention mechanism with an attentional guidance loss for better foreground feature alignment in ViT-based UDA.

Findings

Achieves state-of-the-art results on multiple datasets

Significantly improves domain adaptation performance

Effectively enhances attention focus on task-relevant regions

Abstract

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Recent UDA methods based on Vision Transformers (ViTs) have achieved strong performance through attention-based feature alignment. However, we identify a key limitation: foreground object mismatch, where the discrepancy in foreground object size and spatial distribution across domains weakens attention consistency and hampers effective domain alignment. To address this issue, we propose the Progressive Focus Cross-Attention Mechanism (PCaM), which progressively filters out background information during cross-attention, allowing the model to focus on and fuse discriminative foreground semantics across domains. We further introduce an attentional guidance loss that explicitly directs attention toward task-relevant regions, enhancing cross-domain attention consistency. PCaM is lightweight, architecture-agnostic, and easy to integrate into existing ViT-based UDA pipelines. Extensive experiments on Office-Home, DomainNet, VisDA-2017, and remote sensing datasets demonstrate that PCaM significantly improves adaptation performance and achieves new state-of-the-art results, validating the effectiveness of attention-guided foreground fusion for domain adaptation.