Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground

TL;DR

This paper identifies static bias in video action recognition from both background and foreground elements, and proposes StillMix, a technique that mitigates this bias to improve out-of-distribution generalization.

Contribution

The paper introduces StillMix, a novel bias mitigation method that effectively suppresses static bias from both background and foreground in video data.

Findings

StillMix reduces static bias in video representations.

It improves out-of-distribution generalization in action recognition.

New benchmarks SCUBA and SCUFO evaluate static bias in background and foreground.

Abstract

In video action recognition, shortcut static features can interfere with the learning of motion features, resulting in poor out-of-distribution (OOD) generalization. The video background is clearly a source of static bias, but the video foreground, such as the clothing of the actor, can also provide static bias. In this paper, we empirically verify the existence of foreground static bias by creating test videos with conflicting signals from the static and moving portions of the video. To tackle this issue, we propose a simple yet effective technique, StillMix, to learn robust action representations. Specifically, StillMix identifies bias-inducing video frames using a 2D reference network and mixes them with videos for training, serving as effective bias suppression even when we cannot explicitly extract the source of bias within each video frame or enumerate types of bias. Finally, to precisely evaluate static bias, we synthesize two new benchmarks, SCUBA for static cues in the background, and SCUFO for static cues in the foreground. With extensive experiments, we demonstrate that StillMix mitigates both types of static bias and improves video representations for downstream applications. Code is available at https://github.com/lihaoxin05/StillMix.