Video Domain Incremental Learning for Human Action Recognition in Home Environments

TL;DR

This paper introduces the problem of Video Domain Incremental Learning for human action recognition in home environments, proposing a new benchmark and a replay-based method to address domain shifts without forgetting previous knowledge.

Contribution

It formalizes VDIL, creates a benchmark with three domain split types, and proposes a simple replay-based baseline that outperforms existing methods.

Findings

Replay and reservoir sampling improve continual learning performance.

The proposed method outperforms existing continual learning approaches.

The benchmark reveals significant challenges in domain shifts for home action recognition.

Abstract

It is significantly challenging to recognize daily human actions in homes due to the diversity and dynamic changes in unconstrained home environments. It spurs the need to continually adapt to various users and scenes. Fine-tuning current video understanding models on newly encountered domains often leads to catastrophic forgetting, where the models lose their ability to perform well on previously learned scenarios. To address this issue, we formalize the problem of Video Domain Incremental Learning (VDIL), which enables models to learn continually from different domains while maintaining a fixed set of action classes. Existing continual learning research primarily focuses on class-incremental learning, while the domain incremental learning has been largely overlooked in video understanding. In this work, we introduce a novel benchmark of domain incremental human action recognition for unconstrained home environments. We design three domain split types (user, scene, hybrid) to systematically assess the challenges posed by domain shifts in real-world home settings. Furthermore, we propose a baseline learning strategy based on replay and reservoir sampling techniques without domain labels to handle scenarios with limited memory and task agnosticism. Extensive experimental results demonstrate that our simple sampling and replay strategy outperforms most existing continual learning methods across the three proposed benchmarks.