A Contrastive Objective for Learning Disentangled Representations

TL;DR

This paper introduces a domain-wise contrastive learning approach to create image representations invariant to sensitive attributes, improving bias removal and cross-domain retrieval while balancing invariance and informativeness.

Contribution

The paper proposes a novel domain-wise contrastive objective combined with reconstruction and augmentation techniques to enhance invariant and informative image representations.

Findings

Outperforms state-of-the-art in invariance and informativeness

Effective even without reconstruction constraint

Training is faster and more resource-efficient

Abstract

Learning representations of images that are invariant to sensitive or unwanted attributes is important for many tasks including bias removal and cross domain retrieval. Here, our objective is to learn representations that are invariant to the domain (sensitive attribute) for which labels are provided, while being informative over all other image attributes, which are unlabeled. We present a new approach, proposing a new domain-wise contrastive objective for ensuring invariant representations. This objective crucially restricts negative image pairs to be drawn from the same domain, which enforces domain invariance whereas the standard contrastive objective does not. This domain-wise objective is insufficient on its own as it suffers from shortcut solutions resulting in feature suppression. We overcome this issue by a combination of a reconstruction constraint, image augmentations and initialization with pre-trained weights. Our analysis shows that the choice of augmentations is important, and that a misguided choice of augmentations can harm the invariance and informativeness objectives. In an extensive evaluation, our method convincingly outperforms the state-of-the-art in terms of representation invariance, representation informativeness, and training speed. Furthermore, we find that in some cases our method can achieve excellent results even without the reconstruction constraint, leading to a much faster and resource efficient training.