Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention

TL;DR

This paper introduces CERES, a causal framework that enhances egocentric video object segmentation by addressing dataset biases and visual confounders, leading to improved robustness and state-of-the-art results.

Contribution

The paper proposes a novel causal intervention approach for egocentric video segmentation, integrating dual-modal causal adjustments to improve robustness against biases and confounders.

Findings

CERES achieves state-of-the-art performance on Ego-RVOS benchmarks.

Causal interventions improve robustness to egocentric distortions.

Dual-modal approach effectively mitigates dataset biases and visual confounders.

Abstract

Egocentric Referring Video Object Segmentation (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos. This task is critical for understanding egocentric human behavior. However, achieving such segmentation robustly is challenging due to ambiguities inherent in egocentric videos and biases present in training data. Consequently, existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets and fundamental visual confounding factors of the egocentric perspective, such as rapid motion and frequent occlusions. To address these limitations, we introduce Causal Ego-REferring Segmentation (CERES), a plug-in causal framework that adapts strong, pre-trained RVOS backbones to the egocentric domain. CERES implements dual-modal causal intervention: applying backdoor adjustment principles to counteract language representation biases learned from dataset statistics, and leveraging front-door adjustment concepts to address visual confounding by intelligently integrating semantic visual features with geometric depth information guided by causal principles, creating representations more robust to egocentric distortions. Extensive experiments demonstrate that CERES achieves state-of-the-art performance on Ego-RVOS benchmarks, highlighting the potential of applying causal reasoning to build more reliable models for broader egocentric video understanding.