Discovering environments with XRM
Mohammad Pezeshki, Diane Bouchacourt, Mark Ibrahim, Nicolas Ballas,, Pascal Vincent, David Lopez-Paz
TL;DR
This paper introduces Cross-Risk-Minimization (XRM), an algorithm for automatic environment discovery in datasets that improves out-of-distribution generalization without relying on human annotations or validation sets.
Contribution
XRM is a novel method that trains twin networks to automatically discover environments, eliminating the need for hyper-parameters tuning with human-annotated environments.
Findings
XRM achieves oracle worst-group accuracy in experiments.
XRM does not require early-stopping or validation sets.
Algorithms using XRM environments outperform previous methods.
Abstract
Environment annotations are essential for the success of many out-of-distribution (OOD) generalization methods. Unfortunately, these are costly to obtain and often limited by human annotators' biases. To achieve robust generalization, it is essential to develop algorithms for automatic environment discovery within datasets. Current proposals, which divide examples based on their training error, suffer from one fundamental problem. These methods introduce hyper-parameters and early-stopping criteria, which require a validation set with human-annotated environments, the very information subject to discovery. In this paper, we propose Cross-Risk-Minimization (XRM) to address this issue. XRM trains twin networks, each learning from one random half of the training data, while imitating confident held-out mistakes made by its sibling. XRM provides a recipe for hyper-parameter tuning, does not…
Peer Reviews
Decision·ICML 2024 Oral
1. The paper is well-organized and easy to understand. 2. This paper addresses a crucial challenge in Domain Generalization (DG) tasks, which is the data-splitting process without relying on human annotations. 3. The authors provide strong empirical evidence through extensive experiments to substantiate the effectiveness of their proposed XRM method.
1. The paper's claims may be slightly overstated. While the focus on subpopulation shift in distribution shift is indeed important, it might be more appropriate to avoid claiming to solve a long-standing problem in out-of-distribution generalization without further empirical studies on widely recognized DG benchmarks such as DomainBed and Wilds. These additional experiments could provide more convincing evidence of the proposed approach's effectiveness. 2. The paper lacks a comprehensive discuss
* The problem of learning OOD robust model without manual domain partition is a very important task, which might has great impact on real-world applications. * The proposed method has a clear advantage over existing methods such as EIIL and JTT, that they does not need to explicitly tune the hyperparameter for early stopping. Since hyper-parameter tuning is a crucial challenge, the proposed method would be of interest to many. * The empirical performance is strong.
I have several concerns as follows: 1. The paper should provide a clear discussion on the identifiability challenges presented in [1], which demonstrate that learning invariance without domain partition can be generally impossible. It is crucial to address the need for imposing inductive bias, additional assumptions, conditions, or auxiliary information to ensure the effectiveness of the proposed method. A thorough exploration of these aspects would enhance the paper's theoretical foundation an
- Thorough evaluation on multiple standard datasets. - Good empirical results.
W1. If I understand correctly, the method seems to rely on the fact that misclassified examples are such because they do not contain a "spurious correlation" that a model would learn by default. The twin training serves to reinforce the tendency of one of the trained models to capture this spurious correlation. If this is indeed the case, then the overall methods seems to depend on the (common) heuristic that models learn spurious correlations by default (a.k.a. shortcut learning). I think this
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Anomaly Detection Techniques and Applications · Machine Learning and Data Classification