Deciphering the Projection Head: Representation Evaluation Self-supervised Learning

TL;DR

This paper systematically investigates the role of the projection head in self-supervised learning, revealing its function in promoting semantic feature extraction and proposing a new evaluation design that enhances model robustness and performance.

Contribution

It introduces the Representation Evaluation Design (RED) that improves SSL models by connecting representations and projection vectors, leading to better downstream performance and robustness.

Findings

RED improves baseline SSL models across architectures

Enhanced robustness to unseen augmentations and out-of-distribution data

Consistent performance gains on various datasets

Abstract

Self-supervised learning (SSL) aims to learn intrinsic features without labels. Despite the diverse architectures of SSL methods, the projection head always plays an important role in improving the performance of the downstream task. In this work, we systematically investigate the role of the projection head in SSL. Specifically, the projection head targets the uniformity part of SSL, which pushes the dissimilar samples away from each other, thus enabling the encoder to focus on extracting semantic features. Based on this understanding, we propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built. Extensive experiments with different architectures, including SimCLR, MoCo-V2, and SimSiam, on various datasets, demonstrate that the representation evaluation design can consistently improve the baseline models in the downstream tasks. The learned representation from the RED-SSL models shows superior robustness to unseen augmentations and out-of-distribution data.