When Video Classification Meets Incremental Classes

TL;DR

This paper introduces a novel class-incremental video classification framework that leverages video characteristics and exemplar selection to mitigate catastrophic forgetting, enabling continuous learning of new classes with limited resources.

Contribution

It proposes a new CIVC framework that decomposes spatio-temporal knowledge and uses dual granularity exemplar selection to improve incremental learning performance.

Findings

Outperforms previous SOTA methods on Something-Something V2 and Kinetics datasets.

Effectively alleviates catastrophic forgetting in CIVC.

Demonstrates the benefit of decomposing spatio-temporal knowledge and trajectory-based refinement.

Abstract

With the rapid development of social media, tremendous videos with new classes are generated daily, which raise an urgent demand for video classification methods that can continuously update new classes while maintaining the knowledge of old videos with limited storage and computing resources. In this paper, we summarize this task as Class-Incremental Video Classification (CIVC) and propose a novel framework to address it. As a subarea of incremental learning tasks, the challenge of catastrophic forgetting is unavoidable in CIVC. To better alleviate it, we utilize some characteristics of videos. First, we decompose the spatio-temporal knowledge before distillation rather than treating it as a whole in the knowledge transfer process; trajectory is also used to refine the decomposition. Second, we propose a dual granularity exemplar selection method to select and store representative video instances of old classes and key-frames inside videos under a tight storage budget. We benchmark our method and previous SOTA class-incremental learning methods on Something-Something V2 and Kinetics datasets, and our method outperforms previous methods significantly.