A critical analysis of self-supervision, or what we can learn from a single image

TL;DR

This paper critically examines self-supervision methods for deep learning, revealing that early layers can be learned from a single image with augmentation, but deeper layers require more data and supervision.

Contribution

It demonstrates that early convolutional layers can be learned from a single image using data augmentation, challenging the belief that large datasets are necessary for all layers.

Findings

Early layers can be learned from a single image with augmentation

Deeper layers still require large datasets and manual labels

Low-level image statistics can be captured with synthetic transformations

Abstract

We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.