Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representations

TL;DR

This paper investigates how data augmentations and format transforms like Fourier transforms affect self-supervised learning of image representations, showing that combining them improves classification accuracy and representation quality.

Contribution

It introduces Fourier Domain Augmentations (FDA) and demonstrates their effectiveness alone and with traditional augmentations in enhancing SSL performance.

Findings

Combining FDA with image augmentations improves ImageNet-1K accuracy by up to 1.3%.

Format transforms alone can enhance representation quality without augmentations.

The combination of augmentations and format transforms yields the best results.

Abstract

Self-Supervised Learning (SSL) enables training performant models using limited labeled data. One of the pillars underlying vision SSL is the use of data augmentations/perturbations of the input which do not significantly alter its semantic content. For audio and other temporal signals, augmentations are commonly used alongside format transforms such as Fourier transforms or wavelet transforms. Unlike augmentations, format transforms do not change the information contained in the data; rather, they express the same information in different coordinates. In this paper, we study the effects of format transforms and augmentations both separately and together on vision SSL. We define augmentations in frequency space called Fourier Domain Augmentations (FDA) and show that training SSL models on a combination of these and image augmentations can improve the downstream classification accuracy by up to 1.3% on ImageNet-1K. We also show improvements against SSL baselines in few-shot and transfer learning setups using FDA. Surprisingly, we also observe that format transforms can improve the quality of learned representations even without augmentations; however, the combination of the two techniques yields better quality.