Representation Learning via Invariant Causal Mechanisms

TL;DR

This paper introduces ReLIC, a self-supervised learning method based on invariant causal mechanisms, which improves representation robustness and generalization by enforcing invariance constraints, with strong empirical results on ImageNet and Atari.

Contribution

It proposes ReLIC, a novel self-supervised objective that enforces invariance in representations using causal insights, enhancing robustness and out-of-distribution generalization.

Findings

ReLIC outperforms existing methods on ImageNet in robustness and OOD generalization.

ReLIC achieves above human-level performance on 51 out of 57 Atari games.

Theoretical analysis links contrastive learning success to causal invariance principles.

Abstract

Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using a causal framework. We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose a novel self-supervised objective, Representation Learning via Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning, a particular kind of self-supervised method, and provide an alternative theoretical explanation for the success of these methods. Empirically, ReLIC significantly outperforms competing methods in terms of robustness and out-of-distribution generalization on ImageNet, while also significantly outperforming these methods on Atari achieving above human-level performance on $51$ out of $57$ games.