Meta Curvature-Aware Minimization for Domain Generalization

TL;DR

This paper introduces Meta Curvature-Aware Minimization (MeCAM), a novel training method that encourages models to converge to flatter minima, thereby improving domain generalization performance across multiple benchmark datasets.

Contribution

The paper proposes a new curvature metric and an algorithm, MeCAM, that jointly minimizes training loss, SAM surrogate gap, and meta-learning surrogate gap for better domain generalization.

Findings

MeCAM outperforms existing DG methods on five benchmark datasets.

Theoretical analysis shows improved generalization error and convergence rate.

Empirical results demonstrate superior model robustness and accuracy.

Abstract

Domain generalization (DG) aims to enhance the ability of models trained on source domains to generalize effectively to unseen domains. Recently, Sharpness-Aware Minimization (SAM) has shown promise in this area by reducing the sharpness of the loss landscape to obtain more generalized models. However, SAM and its variants sometimes fail to guide the model toward a flat minimum, and their training processes exhibit limitations, hindering further improvements in model generalization. In this paper, we first propose an improved model training process aimed at encouraging the model to converge to a flat minima. To achieve this, we design a curvature metric that has a minimal effect when the model is far from convergence but becomes increasingly influential in indicating the curvature of the minima as the model approaches a local minimum. Then we derive a novel algorithm from this metric, called Meta Curvature-Aware Minimization (MeCAM), to minimize the curvature around the local minima. Specifically, the optimization objective of MeCAM simultaneously minimizes the regular training loss, the surrogate gap of SAM, and the surrogate gap of meta-learning. We provide theoretical analysis on MeCAM's generalization error and convergence rate, and demonstrate its superiority over existing DG methods through extensive experiments on five benchmark DG datasets, including PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet. Code will be available on GitHub.