Time to augment self-supervised visual representation learning

TL;DR

This paper demonstrates that incorporating time-based augmentations during natural interactions significantly enhances self-supervised visual representation learning, making artificial systems more akin to biological vision.

Contribution

It systematically investigates the benefits of time-based augmentations, revealing their superiority over traditional image augmentations in learning object categories.

Findings

3-D object manipulations improve category learning

Viewing objects against changing backgrounds helps discard background info

Time-based augmentations outperform standard image augmentations

Abstract

Biological vision systems are unparalleled in their ability to learn visual representations without supervision. In machine learning, self-supervised learning (SSL) has led to major advances in forming object representations in an unsupervised fashion. Such systems learn representations invariant to augmentation operations over images, like cropping or flipping. In contrast, biological vision systems exploit the temporal structure of the visual experience during natural interactions with objects. This gives access to "augmentations" not commonly used in SSL, like watching the same object from multiple viewpoints or against different backgrounds. Here, we systematically investigate and compare the potential benefits of such time-based augmentations during natural interactions for learning object categories. Our results show that time-based augmentations achieve large performance gains over state-of-the-art image augmentations. Specifically, our analyses reveal that: 1) 3-D object manipulations drastically improve the learning of object categories; 2) viewing objects against changing backgrounds is important for learning to discard background-related information from the latent representation. Overall, we conclude that time-based augmentations during natural interactions with objects can substantially improve self-supervised learning, narrowing the gap between artificial and biological vision systems.