Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes

TL;DR

This paper introduces a Self-Feedback Training framework that refines loss landscapes to find consistent flat minima across domains, improving out-of-domain generalization in domain generalization tasks.

Contribution

It proposes an iterative method to enhance the consistency of flat minima across domains by refining loss landscapes, leading to better domain generalization performance.

Findings

Outperforms state-of-the-art sharpness-aware methods on multiple benchmarks.

Achieves 2.6% and 1.5% improvements with ResNet-50 and ViT-B/16.

Demonstrates the effectiveness of landscape consistency in domain generalization.

Abstract

Domain generalization aims to learn a model from multiple training domains and generalize it to unseen test domains. Recent theory has shown that seeking the deep models, whose parameters lie in the flat minima of the loss landscape, can significantly reduce the out-of-domain generalization error. However, existing methods often neglect the consistency of loss landscapes in different domains, resulting in models that are not simultaneously in the optimal flat minima in all domains, which limits their generalization ability. To address this issue, this paper proposes an iterative Self-Feedback Training (SFT) framework to seek consistent flat minima that are shared across different domains by progressively refining loss landscapes during training. It alternatively generates a feedback signal by measuring the inconsistency of loss landscapes in different domains and refines these loss landscapes for greater consistency using this feedback signal. Benefiting from the consistency of the flat minima within these refined loss landscapes, our SFT helps achieve better out-of-domain generalization. Extensive experiments on DomainBed demonstrate superior performances of SFT when compared to state-of-the-art sharpness-aware methods and other prevalent DG baselines. On average across five DG benchmarks, SFT surpasses the sharpness-aware minimization by 2.6% with ResNet-50 and 1.5% with ViT-B/16, respectively. The code will be available soon.