Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization

TL;DR

This paper introduces a theoretical framework for domain generalization based on moment alignment, unifying existing approaches and proposing a computationally efficient algorithm that improves generalization across unseen domains.

Contribution

The paper develops a new theory of moment alignment for domain generalization, unifies previous methods, and introduces CMA, a closed-form algorithm that enhances efficiency and effectiveness.

Findings

CMA outperforms ERM and state-of-the-art methods in experiments.

Moment alignment unifies invariant risk minimization, gradient, and Hessian matching.

Theoretical bounds relate domain alignment to transfer measure and generalization.

Abstract

Domain generalization (DG) seeks to develop models that generalize well to unseen target domains, addressing the prevalent issue of distribution shifts in real-world applications. One line of research in DG focuses on aligning domain-level gradients and Hessians to enhance generalization. However, existing methods are computationally inefficient and the underlying principles of these approaches are not well understood. In this paper, we develop the theory of moment alignment for DG. Grounded in \textit{transfer measure}, a principled framework for quantifying generalizability between two domains, we first extend the definition of transfer measure to domain generalization that includes multiple source domains and establish a target error bound. Then, we prove that aligning derivatives across domains improves transfer measure both when the feature extractor induces an invariant optimal predictor across domains and when it does not. Notably, moment alignment provides a unifying understanding of Invariant Risk Minimization, gradient matching, and Hessian matching, three previously disconnected approaches to DG. We further connect feature moments and derivatives of the classifier head, and establish the duality between feature learning and classifier fitting. Building upon our theory, we introduce \textbf{C}losed-Form \textbf{M}oment \textbf{A}lignment (CMA), a novel DG algorithm that aligns domain-level gradients and Hessians in closed-form. Our method overcomes the computational inefficiencies of existing gradient and Hessian-based techniques by eliminating the need for repeated backpropagation or sampling-based Hessian estimation. We validate the efficacy of our approach through two sets of experiments: linear probing and full fine-tuning. CMA demonstrates superior performance in both settings compared to Empirical Risk Minimization and state-of-the-art algorithms.