On the Limitations of General Purpose Domain Generalisation Methods
Henry Gouk, Ondrej Bohdal, Da Li, Timothy Hospedales
TL;DR
This paper analyzes the fundamental limitations of domain generalization methods, showing that standard algorithms like ERM cannot be significantly improved upon in various settings, supported by theoretical bounds and experiments.
Contribution
It provides theoretical bounds on ERM's performance and demonstrates its near-optimality across multiple domain generalization scenarios.
Findings
ERM cannot significantly outperform in DG settings
Theoretical bounds on excess risk of ERM
Experimental validation of theoretical insights
Abstract
We investigate the fundamental performance limitations of learning algorithms in several Domain Generalisation (DG) settings. Motivated by the difficulty with which previously proposed methods have in reliably outperforming Empirical Risk Minimisation (ERM), we derive upper bounds on the excess risk of ERM, and lower bounds on the minimax excess risk. Our findings show that in all the DG settings we consider, it is not possible to significantly outperform ERM. Our conclusions are limited not only to the standard covariate shift setting, but also two other settings with additional restrictions on how domains can differ. The first constrains all domains to have a non-trivial bound on pairwise distances, as measured by a broad class of integral probability metrics. The second alternate setting considers a restricted class of DG problems where all domains have the same underlying support.…
Peer Reviews
Decision·Submitted to ICLR 2025
The paper compiles several structural assumptions of "similarity" between the domains (overlap and IPM distance), and works out the upper and lower bounds on what is statistically achievable in both settings.
On a technical front, the paper is fairly straightforward: since suprema in the Rademacher complexity nicely "split" by triangle inequality, the upper bounds boil down to two application of the standard Rademacher bound machinery. The min-max lower bounds also follow from fairly standard machinery and are fairly expected. This of course would be not a problem if the results were surprising or suggested interesting training interventions --- which unfortunately I don't think it's the case here:
This paper considers the important problem of domain generalization, which is pervasive in modern machine learning. The authors give an interpretation of why ERM performs well under domain generalization setup, which matches the observation in reality.
My main concern is that some of this papers' claim is not rigorous. For example, in the discussion of Theorem 4 (equation (11)), the authors claim that the minimax excess risk scales as $\Omega(\frac{1}{\sqrt{mn}} + \frac{1}{\sqrt{n}})$, since the two terms in lower bound are both $\Omega(\frac{1}{\sqrt{mn}} + \frac{1}{\sqrt{n}})$, and $\Omega(\frac{1}{\sqrt{mn}} + \frac{1}{\sqrt{n}}) - \Omega(\frac{1}{\sqrt{mn}} + \frac{1}{\sqrt{n}})$ scales as $\Omega(\frac{1}{\sqrt{mn}} + \frac{1}{\sqrt{n}})$
This paper presents some theoretical results that quantify the difficulty of generalizing to some unseen domain.
The setup of this paper is quite broad, with very general theorems, making it challenging to assess the precise difficulty of the domain generalization problem. One popular approach in domain generalization is invariant risk minimization (IRM), introduced by Arjovsky et al., and subsequently explored in other studies. This line of research aims to identify patterns that are consistent across multiple domains, thereby disregarding spurious correlations that vary across domains. This approach alig
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Topic Modeling · Machine Learning in Healthcare