Simulate, Refocus and Ensemble: An Attention-Refocusing Scheme for Domain Generalization

TL;DR

This paper introduces SRE, a novel attention-refocusing scheme that enhances CLIP's domain generalization by simulating domain shifts, refocusing attention, and ensemble learning to improve performance on unseen domains.

Contribution

The paper proposes a new attention-refocusing scheme called SRE that aligns CLIP's attention maps across simulated and real target domains for better generalization.

Findings

SRE outperforms state-of-the-art methods on multiple datasets.

Attention refocusing improves domain-invariant feature learning.

Simulating domain shifts enhances model robustness.

Abstract

Domain generalization (DG) aims to learn a model from source domains and apply it to unseen target domains with out-of-distribution data. Owing to CLIP's strong ability to encode semantic concepts, it has attracted increasing interest in domain generalization. However, CLIP often struggles to focus on task-relevant regions across domains, i.e., domain-invariant regions, resulting in suboptimal performance on unseen target domains. To address this challenge, we propose an attention-refocusing scheme, called Simulate, Refocus and Ensemble (SRE), which learns to reduce the domain shift by aligning the attention maps in CLIP via attention refocusing. SRE first simulates domain shifts by performing augmentation on the source data to generate simulated target domains. SRE then learns to reduce the domain shifts by refocusing the attention in CLIP between the source and simulated target domains. Finally, SRE utilizes ensemble learning to enhance the ability to capture domain-invariant attention maps between the source data and the simulated target data. Extensive experimental results on several datasets demonstrate that SRE generally achieves better results than state-of-the-art methods. The code is available at: https://github.com/bitPrincy/SRE-DG.