SOAR: Scene-debiasing Open-set Action Recognition

TL;DR

SOAR introduces adversarial modules to reduce scene bias in open-set action recognition, leading to improved accuracy and robustness across different scene distributions.

Contribution

The paper proposes a novel scene-debiasing framework with adversarial modules to mitigate scene bias in open-set action recognition tasks.

Findings

SOAR reduces scene bias in action recognition.

The method outperforms state-of-the-art approaches.

Ablation studies validate the effectiveness of the modules.

Abstract

Deep learning models have a risk of utilizing spurious clues to make predictions, such as recognizing actions based on the background scene. This issue can severely degrade the open-set action recognition performance when the testing samples have different scene distributions from the training samples. To mitigate this problem, we propose a novel method, called Scene-debiasing Open-set Action Recognition (SOAR), which features an adversarial scene reconstruction module and an adaptive adversarial scene classification module. The former prevents the decoder from reconstructing the video background given video features, and thus helps reduce the background information in feature learning. The latter aims to confuse scene type classification given video features, with a specific emphasis on the action foreground, and helps to learn scene-invariant information. In addition, we design an experiment to quantify the scene bias. The results indicate that the current open-set action recognizers are biased toward the scene, and our proposed SOAR method better mitigates such bias. Furthermore, our extensive experiments demonstrate that our method outperforms state-of-the-art methods, and the ablation studies confirm the effectiveness of our proposed modules.