Lifelong Learning of Video Diffusion Models From a Single Video Stream

TL;DR

This paper shows that autoregressive video diffusion models can be trained effectively from a single video stream using experience replay, enabling lifelong learning in streaming environments with new datasets.

Contribution

It introduces a method for lifelong learning of video diffusion models from a single stream and provides four new datasets for streaming generative video modeling.

Findings

Training from a single video stream is as effective as offline training.

Experience replay with a subset of data suffices for lifelong learning.

Four new datasets support streaming lifelong generative video modeling.

Abstract

This work demonstrates that training autoregressive video diffusion models from a single video stream$\unicode{x2013}$resembling the experience of embodied agents$\unicode{x2013}$is not only possible, but can also be as effective as standard offline training given the same number of gradient steps. Our work further reveals that this main result can be achieved using experience replay methods that only retain a subset of the preceding video stream. To support training and evaluation in this setting, we introduce four new datasets for streaming lifelong generative video modeling: Lifelong Bouncing Balls, Lifelong 3D Maze, Lifelong Drive, and Lifelong PLAICraft, each consisting of one million consecutive frames from environments of increasing complexity.