Continual Learning of Predictive Models in Video Sequences via Variational Autoencoders

TL;DR

This paper introduces a continual learning approach for video prediction using Variational Autoencoders and an adapted Markov Jump Particle Filter to recognize new situations and prevent forgetting.

Contribution

It presents a novel method combining VAEs and a specialized particle filter for continual video prediction learning without catastrophic forgetting.

Findings

Effective recognition of new situations in video sequences

Prevents catastrophic forgetting in continual learning

Successful application in vehicle task sequences

Abstract

This paper proposes a method for performing continual learning of predictive models that facilitate the inference of future frames in video sequences. For a first given experience, an initial Variational Autoencoder, together with a set of fully connected neural networks are utilized to respectively learn the appearance of video frames and their dynamics at the latent space level. By employing an adapted Markov Jump Particle Filter, the proposed method recognizes new situations and integrates them as predictive models avoiding catastrophic forgetting of previously learned tasks. For evaluating the proposed method, this article uses video sequences from a vehicle that performs different tasks in a controlled environment.