OccludeNet: A Causal Journey into Mixed-View Actor-Centric Video Action Recognition under Occlusions

TL;DR

This paper introduces OccludeNet, a large-scale occluded video dataset with real and synthetic occlusions, and proposes a causal model-based method to improve action recognition robustness under occlusion.

Contribution

It provides a new dataset with diverse occlusion scenarios and develops a causal inference approach for more robust action recognition in occluded scenes.

Findings

Occlusion impacts different action classes variably.

The causal model improves recognition accuracy under occlusion.

The dataset enables studying causal links in occluded scenes.

Abstract

The lack of occlusion data in common action recognition video datasets limits model robustness and hinders consistent performance gains. We build OccludeNet, a large-scale occluded video dataset including both real and synthetic occlusion scenes in different natural settings. OccludeNet includes dynamic occlusion, static occlusion, and multi-view interactive occlusion, addressing gaps in current datasets. Our analysis shows occlusion affects action classes differently: actions with low scene relevance and partial body visibility see larger drops in accuracy. To overcome the limits of existing occlusion-aware methods, we propose a structural causal model for occluded scenes and introduce the Causal Action Recognition (CAR) method, which uses backdoor adjustment and counterfactual reasoning. This approach strengthens key actor information and improves model robustness to occlusion. We hope the challenges of OccludeNet will encourage more study of causal links in occluded scenes and lead to a fresh look at class relations, ultimately leading to lasting performance improvements. Our code and data is availibale at: https://github.com/The-Martyr/OccludeNet-Dataset