Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams

TL;DR

This paper introduces a neural network approach with attention and stochastic coherence for unsupervised, pixel-wise representation learning in video streams, enabling open-set class-incremental classification with minimal supervision.

Contribution

It proposes a novel attention-based, unsupervised learning method using stochastic coherence for continuous, pixel-level video understanding in an open-set setting.

Findings

Agents can learn to distinguish objects from video streams.

The method outperforms state-of-the-art models in open-set classification.

Unsupervised learning with attention and coherence is effective in dynamic environments.

Abstract

Devising intelligent agents able to live in an environment and learn by observing the surroundings is a longstanding goal of Artificial Intelligence. From a bare Machine Learning perspective, challenges arise when the agent is prevented from leveraging large fully-annotated dataset, but rather the interactions with supervisory signals are sparsely distributed over space and time. This paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Spatio-temporal stochastic coherence along the attention trajectory, paired with a contrastive term, leads to an unsupervised learning criterion that naturally copes with the considered setting. Differently from most existing works, the learned representations are used in open-set class-incremental classification of each frame pixel, relying on few supervisions. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream. Inheriting features from state-of-the art models is not as powerful as one might expect.