Risk Variance Penalization

TL;DR

This paper introduces Risk Variance Penalization (RVP), a theoretically justified modification of V-REx, which enhances out-of-distribution generalization by discovering invariant predictors through a variance-based regularizer.

Contribution

The paper provides theoretical insights into V-REx, proposes RVP as a modified regularizer with theory-based tuning, and demonstrates its effectiveness in invariant predictor discovery.

Findings

RVP addresses theoretical concerns of V-REx.

RVP can find invariant predictors under certain conditions.

Experimental results validate RVP's robustness and invariance discovery.

Abstract

The key of the out-of-distribution (OOD) generalization is to generalize invariance from training domains to target domains. The variance risk extrapolation (V-REx) is a practical OOD method, which depends on a domain-level regularization but lacks theoretical verifications about its motivation and utility. This article provides theoretical insights into V-REx by studying a variance-based regularizer. We propose Risk Variance Penalization (RVP), which slightly changes the regularization of V-REx but addresses the theory concerns about V-REx. We provide theoretical explanations and a theory-inspired tuning scheme for the regularization parameter of RVP. Our results point out that RVP discovers a robust predictor. Finally, we experimentally show that the proposed regularizer can find an invariant predictor under certain conditions.