Seeing Beyond the Scene: Analyzing and Mitigating Background Bias in Action Recognition

TL;DR

This paper investigates background bias in action recognition models, analyzes its prevalence across various models, and proposes mitigation strategies including segmented human input and prompt tuning to enhance human-focused reasoning.

Contribution

It provides a systematic analysis of background bias in multiple models and introduces effective mitigation techniques for both classification models and VLLMs.

Findings

Background bias is prevalent across models.

Segmented human input reduces background bias by 3.78%.

Prompt tuning steers VLLMs towards human-focused reasoning by 9.85%.

Abstract

Human action recognition models often rely on background cues rather than human movement and pose to make predictions, a behavior known as background bias. We present a systematic analysis of background bias across classification models, contrastive text-image pretrained models, and Video Large Language Models (VLLM) and find that all exhibit a strong tendency to default to background reasoning. Next, we propose mitigation strategies for classification models and show that incorporating segmented human input effectively decreases background bias by 3.78%. Finally, we explore manual and automated prompt tuning for VLLMs, demonstrating that prompt design can steer predictions towards human-focused reasoning by 9.85%.