Using Navigational Information to Learn Visual Representations

TL;DR

This paper demonstrates that incorporating navigational spatial and temporal information into contrastive learning enhances the quality of visual representations, surpassing traditional instance discrimination methods in downstream classification tasks.

Contribution

It introduces a novel pretraining pipeline that leverages self-generated navigational data in a photorealistic environment to improve self-supervised visual learning.

Findings

Spatial and temporal info improves representation quality

Enhanced downstream classification performance

Contextual info is more effective than instance discrimination

Abstract

Children learn to build a visual representation of the world from unsupervised exploration and we hypothesize that a key part of this learning ability is the use of self-generated navigational information as a similarity label to drive a learning objective for self-supervised learning. The goal of this work is to exploit navigational information in a visual environment to provide performance in training that exceeds the state-of-the-art self-supervised training. Here, we show that using spatial and temporal information in the pretraining stage of contrastive learning can improve the performance of downstream classification relative to conventional contrastive learning approaches that use instance discrimination to discriminate between two alterations of the same image or two different images. We designed a pipeline to generate egocentric-vision images from a photorealistic ray-tracing environment (ThreeDWorld) and record relevant navigational information for each image. Modifying the Momentum Contrast (MoCo) model, we introduced spatial and temporal information to evaluate the similarity of two views in the pretraining stage instead of instance discrimination. This work reveals the effectiveness and efficiency of contextual information for improving representation learning. The work informs our understanding of the means by which children might learn to see the world without external supervision.