Causal Transportability for Visual Recognition

TL;DR

This paper introduces a causal inference framework for visual recognition that improves out-of-distribution generalization by estimating invariant causal effects, addressing spurious correlations in classifiers.

Contribution

It develops an algorithm to estimate causal effects in image classification that remains invariant across different environments, enhancing robustness.

Findings

The approach captures causal invariances in visual data.

It improves out-of-distribution generalization performance.

Theoretical and empirical results validate the method's effectiveness.

Abstract

Visual representations underlie object recognition tasks, but they often contain both robust and non-robust features. Our main observation is that image classifiers may perform poorly on out-of-distribution samples because spurious correlations between non-robust features and labels can be changed in a new environment. By analyzing procedures for out-of-distribution generalization with a causal graph, we show that standard classifiers fail because the association between images and labels is not transportable across settings. However, we then show that the causal effect, which severs all sources of confounding, remains invariant across domains. This motivates us to develop an algorithm to estimate the causal effect for image classification, which is transportable (i.e., invariant) across source and target environments. Without observing additional variables, we show that we can derive an estimand for the causal effect under empirical assumptions using representations in deep models as proxies. Theoretical analysis, empirical results, and visualizations show that our approach captures causal invariances and improves overall generalization.