Breaking Correlation Shift via Conditional Invariant Regularizer

TL;DR

This paper introduces a new regularizer based on conditional independence to improve out-of-distribution generalization under correlation shift, supported by a novel metric and a provably convergent algorithm.

Contribution

It proposes the Conditional Spurious Variation (CSV) metric and a regularization method to enhance OOD generalization by enforcing conditional independence from spurious attributes.

Findings

The proposed method improves OOD generalization in experiments.

The CSV metric effectively measures conditional independence.

The algorithm converges with a provable rate.

Abstract

Recently, generalization on out-of-distribution (OOD) data with correlation shift has attracted great attentions. The correlation shift is caused by the spurious attributes that correlate to the class label, as the correlation between them may vary in training and test data. For such a problem, we show that given the class label, the models that are conditionally independent of spurious attributes are OOD generalizable. Based on this, a metric Conditional Spurious Variation (CSV) which controls the OOD generalization error, is proposed to measure such conditional independence. To improve the OOD generalization, we regularize the training process with the proposed CSV. Under mild assumptions, our training objective can be formulated as a nonconvex-concave mini-max problem. An algorithm with a provable convergence rate is proposed to solve the problem. Extensive empirical results verify our algorithm's efficacy in improving OOD generalization.