Learning Gradient-based Mixup with Extrapolation toward Flatter Minima for Domain Generalization

TL;DR

This paper introduces FGMix, a gradient-based mixup method with extrapolation aimed at covering unseen data regions and finding flatter minima to improve domain generalization performance.

Contribution

The paper proposes a novel mixup policy that uses gradient compatibility to generate invariant features and encourages flatter minima for better unseen domain generalization.

Findings

FGMix outperforms existing DG methods on DomainBed benchmark.

Gradient-based mixup with extrapolation enhances coverage of unseen regions.

Flatter minima correlate with improved domain generalization.

Abstract

To address distribution shifts between training and test data, domain generalization (DG) leverages multiple source domains to learn a model that generalizes well to unseen domains. However, existing DG methods often overfit to the source domains, partly due to the limited coverage of the expected region in feature space. Motivated by this, we propose performing mixup with data interpolation and extrapolation to cover potentially unseen regions. To prevent the detrimental effects of unconstrained extrapolation, we carefully design a policy to generate the instance weights, named Flatness-aware Gradient-based Mixup (FGMix). The policy relies on gradient-based compatibilities to assign greater weights to instances that carry more invariant information and learn the mixup policy towards flatter minima for better generalization. On the DomainBed benchmark, we validate the efficacy of various designs of FGMix and demonstrate its superiority over other DG algorithms.