Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows

TL;DR

This paper introduces a novel unsupervised continual learning approach for pixel-wise visual features that leverages autonomous, multi-level motion flows to improve representation consistency, outperforming existing methods on synthetic and real videos.

Contribution

It proposes a self-supervised method that learns higher-order motion flows without external signals, enhancing continual visual feature learning.

Findings

Outperforms state-of-the-art unsupervised models on synthetic and real videos.

Develops multi-level motion flows from optical to higher-order signals.

Uses contrastive loss to maintain feature consistency during learning.

Abstract

Learning with neural networks from a continuous stream of visual information presents several challenges due to the non-i.i.d. nature of the data. However, it also offers novel opportunities to develop representations that are consistent with the information flow. In this paper we investigate the case of unsupervised continual learning of pixel-wise features subject to multiple motion-induced constraints, therefore named motion-conjugated feature representations. Differently from existing approaches, motion is not a given signal (either ground-truth or estimated by external modules), but is the outcome of a progressive and autonomous learning process, occurring at various levels of the feature hierarchy. Multiple motion flows are estimated with neural networks and characterized by different levels of abstractions, spanning from traditional optical flow to other latent signals originating from higher-level features, hence called higher-order motions. Continuously learning to develop consistent multi-order flows and representations is prone to trivial solutions, which we counteract by introducing a self-supervised contrastive loss, spatially-aware and based on flow-induced similarity. We assess our model on photorealistic synthetic streams and real-world videos, comparing to pre-trained state-of-the art feature extractors (also based on Transformers) and to recent unsupervised learning models, significantly outperforming these alternatives.