Stable and Causal Inference for Discriminative Self-supervised Deep Visual Representations

TL;DR

This paper analyzes the instability of discriminative self-supervised visual representations from a causal perspective and proposes inference-time solutions involving synthetic data to improve stability and downstream performance.

Contribution

It introduces a causal analysis of self-supervised methods and proposes inference-time interventions using synthetic data to enhance stability and efficiency.

Findings

Proposed solutions improve representation stability.

Inference-time methods outperform training-time approaches.

Effective on both controlled and real-world datasets.

Abstract

In recent years, discriminative self-supervised methods have made significant strides in advancing various visual tasks. The central idea of learning a data encoder that is robust to data distortions/augmentations is straightforward yet highly effective. Although many studies have demonstrated the empirical success of various learning methods, the resulting learned representations can exhibit instability and hinder downstream performance. In this study, we analyze discriminative self-supervised methods from a causal perspective to explain these unstable behaviors and propose solutions to overcome them. Our approach draws inspiration from prior works that empirically demonstrate the ability of discriminative self-supervised methods to demix ground truth causal sources to some extent. Unlike previous work on causality-empowered representation learning, we do not apply our solutions during the training process but rather during the inference process to improve time efficiency. Through experiments on both controlled image datasets and realistic image datasets, we show that our proposed solutions, which involve tempering a linear transformation with controlled synthetic data, are effective in addressing these issues.