GUESS: Generative Uncertainty Ensemble for Self Supervision

TL;DR

GUESS introduces a novel SSL framework that incorporates uncertainty representation and ensemble methods to improve data-dependent invariance enforcement, leading to more robust feature learning from unlabeled data.

Contribution

The paper proposes GUESS, a new SSL approach combining uncertainty modeling, a pseudo-whitening architecture, and a generative-discriminative loss for enhanced invariant feature learning.

Findings

GUESS outperforms existing SSL methods on benchmark datasets.

Uncertainty injection improves robustness of learned representations.

Ablation studies confirm the effectiveness of each component.

Abstract

Self-supervised learning (SSL) frameworks consist of pretext task, and loss function aiming to learn useful general features from unlabeled data. The basic idea of most SSL baselines revolves around enforcing the invariance to a variety of data augmentations via the loss function. However, one main issue is that, inattentive or deterministic enforcement of the invariance to any kind of data augmentation is generally not only inefficient, but also potentially detrimental to performance on the downstream tasks. In this work, we investigate the issue from the viewpoint of uncertainty in invariance representation. Uncertainty representation is fairly under-explored in the design of SSL architectures as well as loss functions. We incorporate uncertainty representation in both loss function as well as architecture design aiming for more data-dependent invariance enforcement. The former is represented in the form of data-derived uncertainty in SSL loss function resulting in a generative-discriminative loss function. The latter is achieved by feeding slightly different distorted versions of samples to the ensemble aiming for learning better and more robust representation. Specifically, building upon the recent methods that use hard and soft whitening (a.k.a redundancy reduction), we introduce a new approach GUESS, a pseudo-whitening framework, composed of controlled uncertainty injection, a new architecture, and a new loss function. We include detailed results and ablation analysis establishing GUESS as a new baseline.