Survey on Self-supervised Representation Learning Using Image Transformations

TL;DR

This survey reviews self-supervised learning methods based on geometric transformations, highlighting their architectures, performance on image recognition tasks, and potential for future research in unsupervised representation learning.

Contribution

It provides a focused review of SSL techniques using geometric transformations, including model architectures, performance analysis, and insights for future work.

Findings

AETv2 outperforms other models on CIFAR-10 and ImageNet

Rotation with feature decoupling performs well in certain settings

Geometric transformations are effective supervisory signals in SSL

Abstract

Deep neural networks need huge amount of training data, while in real world there is a scarcity of data available for training purposes. To resolve these issues, self-supervised learning (SSL) methods are used. SSL using geometric transformations (GT) is a simple yet powerful technique used in unsupervised representation learning. Although multiple survey papers have reviewed SSL techniques, there is none that only focuses on those that use geometric transformations. Furthermore, such methods have not been covered in depth in papers where they are reviewed. Our motivation to present this work is that geometric transformations have shown to be powerful supervisory signals in unsupervised representation learning. Moreover, many such works have found tremendous success, but have not gained much attention. We present a concise survey of SSL approaches that use geometric transformations. We shortlist six representative models that use image transformations including those based on predicting and autoencoding transformations. We review their architecture as well as learning methodologies. We also compare the performance of these models in the object recognition task on CIFAR-10 and ImageNet datasets. Our analysis indicates the AETv2 performs the best in most settings. Rotation with feature decoupling also performed well in some settings. We then derive insights from the observed results. Finally, we conclude with a summary of the results and insights as well as highlighting open problems to be addressed and indicating various future directions.