Automatic Shortcut Removal for Self-Supervised Representation Learning

TL;DR

This paper introduces a framework to identify and mitigate shortcut features in self-supervised visual learning by using an adversarial lens network, leading to improved semantic representations.

Contribution

It proposes a novel adversarial approach to reduce shortcut reliance in self-supervised learning, enhancing the quality of learned representations.

Findings

Modified images improve representation quality across datasets

The lens reveals how pretext tasks influence learned features

Shortcut features are effectively mitigated by the proposed method

Abstract

In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for mitigating the effect shortcut features. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.