RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank

TL;DR

RankMe introduces an unsupervised, label-free criterion based on the effective rank of representations to predict downstream performance of self-supervised learning models, aiding deployment without labels.

Contribution

The paper proposes RankMe, a simple, hyper-parameter-free method to assess JE-SSL representations' quality using their effective rank, enabling performance prediction without labels.

Findings

RankMe accurately predicts downstream performance across datasets.

It facilitates hyperparameter tuning with minimal performance loss.

It is computationally efficient and easy to deploy.

Abstract

Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations but only few principled guidelines that would help practitioners to successfully deploy them. The main reason for that pitfall comes from JE-SSL's core principle of not employing any input reconstruction therefore lacking visual cues of unsuccessful training. Adding non informative loss values to that, it becomes difficult to deploy SSL on a new dataset for which no labels can help to judge the quality of the learned representation. In this study, we develop a simple unsupervised criterion that is indicative of the quality of the learned JE-SSL representations: their effective rank. Albeit simple and computationally friendly, this method -- coined RankMe -- allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels. A further benefit of RankMe is that it does not have any training or hyper-parameters to tune. Through thorough empirical experiments involving hundreds of training episodes, we demonstrate how RankMe can be used for hyperparameter selection with nearly no reduction in final performance compared to the current selection method that involve a dataset's labels. We hope that RankMe will facilitate the deployment of JE-SSL towards domains that do not have the opportunity to rely on labels for representations' quality assessment.