ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning

TL;DR

This paper introduces ESSENTIAL, a novel memory integration approach for video class-incremental learning that combines episodic and semantic memory to improve performance while reducing memory usage.

Contribution

The paper proposes a new memory retrieval module that integrates episodic and semantic memories via cross-attention, enhancing video class-incremental learning efficiency.

Findings

Achieves competitive performance with less memory on multiple datasets.

Effectively retrieves dense features from sparse memory representations.

Outperforms existing methods in memory efficiency and accuracy.

Abstract

In this work, we tackle the problem of video classincremental learning (VCIL). Many existing VCIL methods mitigate catastrophic forgetting by rehearsal training with a few temporally dense samples stored in episodic memory, which is memory-inefficient. Alternatively, some methods store temporally sparse samples, sacrificing essential temporal information and thereby resulting in inferior performance. To address this trade-off between memory-efficiency and performance, we propose EpiSodic and SEmaNTIc memory integrAtion for video class-incremental Learning (ESSENTIAL). ESSENTIAL consists of episodic memory for storing temporally sparse features and semantic memory for storing general knowledge represented by learnable prompts. We introduce a novel memory retrieval (MR) module that integrates episodic memory and semantic prompts through cross-attention, enabling the retrieval of temporally dense features from temporally sparse features. We rigorously validate ESSENTIAL on diverse datasets: UCF-101, HMDB51, and Something-Something-V2 from the TCD benchmark and UCF-101, ActivityNet, and Kinetics-400 from the vCLIMB benchmark. Remarkably, with significantly reduced memory, ESSENTIAL achieves favorable performance on the benchmarks.