Continual Predictive Learning from Videos

TL;DR

This paper introduces a continual learning framework for video prediction in non-stationary environments, addressing catastrophic forgetting with a mixture world model and test-time adaptation, demonstrated on RoboNet and KTH benchmarks.

Contribution

The paper proposes the continual predictive learning (CPL) method combining predictive experience replay and non-parametric task inference for non-stationary video prediction environments.

Findings

CPL effectively mitigates catastrophic forgetting.

CPL outperforms naive combinations of existing methods.

New benchmarks based on RoboNet and KTH are introduced.

Abstract

Predictive learning ideally builds the world model of physical processes in one or more given environments. Typical setups assume that we can collect data from all environments at all times. In practice, however, different prediction tasks may arrive sequentially so that the environments may change persistently throughout the training procedure. Can we develop predictive learning algorithms that can deal with more realistic, non-stationary physical environments? In this paper, we study a new continual learning problem in the context of video prediction, and observe that most existing methods suffer from severe catastrophic forgetting in this setup. To tackle this problem, we propose the continual predictive learning (CPL) approach, which learns a mixture world model via predictive experience replay and performs test-time adaptation with non-parametric task inference. We construct two new benchmarks based on RoboNet and KTH, in which different tasks correspond to different physical robotic environments or human actions. Our approach is shown to effectively mitigate forgetting and remarkably outperform the na\"ive combinations of previous art in video prediction and continual learning.