Generalizable Information Theoretic Causal Representation

TL;DR

This paper introduces a causal representation learning method using mutual information regularization guided by a hypothetical causal graph, leading to improved generalization and robustness in downstream tasks.

Contribution

It proposes a novel causal representation learning approach with theoretical guarantees and empirical validation demonstrating enhanced robustness and generalization.

Findings

Reduced sample complexity in causal learning

Improved robustness against adversarial attacks

Enhanced generalization under distribution shifts

Abstract

It is evidence that representation learning can improve model's performance over multiple downstream tasks in many real-world scenarios, such as image classification and recommender systems. Existing learning approaches rely on establishing the correlation (or its proxy) between features and the downstream task (labels), which typically results in a representation containing cause, effect and spurious correlated variables of the label. Its generalizability may deteriorate because of the unstability of the non-causal parts. In this paper, we propose to learn causal representation from observational data by regularizing the learning procedure with mutual information measures according to our hypothetical causal graph. The optimization involves a counterfactual loss, based on which we deduce a theoretical guarantee that the causality-inspired learning is with reduced sample complexity and better generalization ability. Extensive experiments show that the models trained on causal representations learned by our approach is robust under adversarial attacks and distribution shift.