Learning predictable and robust neural representations by straightening image sequences

TL;DR

This paper introduces a self-supervised learning method that encourages neural representations to follow straighter temporal trajectories, leading to more predictive, robust, and disentangled features in image sequence models.

Contribution

The authors propose a novel straightening objective for SSL that improves predictability, robustness, and attribute disentanglement in neural representations, inspired by primate visual system findings.

Findings

Learned embeddings are predictive and disentangle object attributes.

Models trained with straightening are more noise- and adversarial-robust.

Straightening can be used as a regularizer for other training methods.

Abstract

Prediction is a fundamental capability of all living organisms, and has been proposed as an objective for learning sensory representations. Recent work demonstrates that in primate visual systems, prediction is facilitated by neural representations that follow straighter temporal trajectories than their initial photoreceptor encoding, which allows for prediction by linear extrapolation. Inspired by these experimental findings, we develop a self-supervised learning (SSL) objective that explicitly quantifies and promotes straightening. We demonstrate the power of this objective in training deep feedforward neural networks on smoothly-rendered synthetic image sequences that mimic commonly-occurring properties of natural videos. The learned model contains neural embeddings that are predictive, but also factorize the geometric, photometric, and semantic attributes of objects. The representations also prove more robust to noise and adversarial attacks compared to previous SSL methods that optimize for invariance to random augmentations. Moreover, these beneficial properties can be transferred to other training procedures by using the straightening objective as a regularizer, suggesting a broader utility for straightening as a principle for robust unsupervised learning.