Effect of Rotation Angle in Self-Supervised Pre-training is Dataset-Dependent

TL;DR

This paper investigates how rotation angles in contrastive self-supervised pre-training affect feature learning, revealing that the impact varies across datasets and may relate to gradient orientation distributions.

Contribution

It demonstrates that the effect of rotation angles in self-supervised pre-training is dataset-dependent and explores potential underlying factors like gradient orientation distributions.

Findings

Rotation angle effects vary across datasets.

Saliency maps show dataset-dependent feature focus.

Gradient distribution may influence rotation angle impact.

Abstract

Self-supervised learning for pre-training (SSP) can help the network learn better low-level features, especially when the size of the training set is small. In contrastive pre-training, the network is pre-trained to distinguish between different versions of the input. For example, the network learns to distinguish pairs (original, rotated) of images where the rotated image was rotated by angle $\theta$ vs. other pairs of images. In this work, we show that, when training using contrastive pre-training in this way, the angle $\theta$ and the dataset interact in interesting ways. We hypothesize, and give some evidence, that, for some datasets, the network can take "shortcuts" for particular rotation angles $\theta$ based on the distribution of the gradient directions in the input, possibly avoiding learning features other than edges, but our experiments do not seem to support that hypothesis. We demonstrate experiments on three radiology datasets. We compute the saliency map indicating which pixels were important in the SSP process, and compare the saliency map to the ground truth foreground/background segmentation. Our visualizations indicate that the effects of rotation angles in SSP are dataset-dependent. We believe the distribution of gradient orientations may play a role in this, but our experiments so far are inconclusive.