Class-Incremental Learning for Action Recognition in Videos

TL;DR

This paper presents a novel framework for class-incremental video action recognition that mitigates catastrophic forgetting using importance maps, knowledge distillation, and feature decorrelation, demonstrating improved performance on standard benchmarks.

Contribution

It introduces time-channel importance maps and a regularization scheme for continual learning in video recognition, addressing a gap in existing methods.

Findings

Outperforms existing continual learning methods on UCF101, HMDB51, and Something-Something V2 datasets.

Effectively reduces catastrophic forgetting in incremental video action recognition.

Demonstrates the importance of importance maps and feature decorrelation in continual learning.

Abstract

We tackle catastrophic forgetting problem in the context of class-incremental learning for video recognition, which has not been explored actively despite the popularity of continual learning. Our framework addresses this challenging task by introducing time-channel importance maps and exploiting the importance maps for learning the representations of incoming examples via knowledge distillation. We also incorporate a regularization scheme in our objective function, which encourages individual features obtained from different time steps in a video to be uncorrelated and eventually improves accuracy by alleviating catastrophic forgetting. We evaluate the proposed approach on brand-new splits of class-incremental action recognition benchmarks constructed upon the UCF101, HMDB51, and Something-Something V2 datasets, and demonstrate the effectiveness of our algorithm in comparison to the existing continual learning methods that are originally designed for image data.