Causality-oriented robustness: exploiting general noise interventions

TL;DR

This paper introduces DRIG, a causality-based method leveraging noise interventions to create robust prediction models that generalize well across unseen distribution shifts, bridging in-distribution accuracy and causal invariance.

Contribution

The paper proposes DRIG, a novel approach exploiting general noise interventions for robust predictions, and demonstrates its theoretical and empirical advantages over existing methods.

Findings

DRIG provides robustness against diverse distribution shifts.

It includes anchor regression as a special case.

Empirical results show improved performance on real datasets.

Abstract

Since distribution shifts are common in real-world applications, there is a pressing need to develop prediction models that are robust against such shifts. Existing frameworks, such as empirical risk minimization or distributionally robust optimization, either lack generalizability for unseen distributions or rely on postulated distance measures. Alternatively, causality offers a data-driven and structural perspective to robust predictions. However, the assumptions necessary for causal inference can be overly stringent, and the robustness offered by such causal models often lacks flexibility. In this paper, we focus on causality-oriented robustness and propose Distributional Robustness via Invariant Gradients (DRIG), a method that exploits general noise interventions in training data for robust predictions against unseen interventions, and naturally interpolates between in-distribution prediction and causality. In a linear setting, we prove that DRIG yields predictions that are robust among a data-dependent class of distribution shifts. Furthermore, we show that our framework includes anchor regression as a special case, and that it yields prediction models that protect against more diverse perturbations. We establish finite-sample results and extend our approach to semi-supervised domain adaptation to further improve prediction performance. Finally, we empirically validate our methods on synthetic simulations and on single-cell and intensive health care datasets.