ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition

TL;DR

ALBAR introduces an adversarial training method that effectively reduces both foreground and background biases in action recognition models, improving fairness and accuracy without needing bias-specific annotations.

Contribution

The paper presents a novel adversarial learning framework for mitigating action recognition biases, addressing both foreground and background biases without specialized knowledge of bias attributes.

Findings

Sets new state-of-the-art debiasing performance on HMDB51.

Improves combined debiasing accuracy by over 12% absolute.

Identifies and addresses background leakage issues in existing bias evaluation protocols.

Abstract

Bias in machine learning models can lead to unfair decision making, and while it has been well-studied in the image and text domains, it remains underexplored in action recognition. Action recognition models often suffer from background bias (i.e., inferring actions based on background cues) and foreground bias (i.e., relying on subject appearance), which can be detrimental to real-life applications such as autonomous vehicles or assisted living monitoring. While prior approaches have mainly focused on mitigating background bias using specialized augmentations, we thoroughly study both foreground and background bias. We propose ALBAR, a novel adversarial training method that mitigates foreground and background biases without requiring specialized knowledge of the bias attributes. Our framework applies an adversarial cross-entropy loss to the sampled static clip (where all the frames are the same) and aims to make its class probabilities uniform using a proposed entropy maximization loss. Additionally, we introduce a gradient penalty loss for regularization against the debiasing process. We evaluate our method on established background and foreground bias protocols, setting a new state-of-the-art and strongly improving combined debiasing performance by over 12% absolute on HMDB51. Furthermore, we identify an issue of background leakage in the existing UCF101 protocol for bias evaluation which provides a shortcut to predict actions and does not provide an accurate measure of the debiasing capability of a model. We address this issue by proposing more fine-grained segmentation boundaries for the actor, where our method also outperforms existing approaches. Project Page: https://joefioresi718.github.io/ALBAR_webpage/