DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering

TL;DR

This paper introduces DMC$^3$, a novel framework for egocentric video question answering that leverages counterfactual contrastive learning to better understand first-person videos and improve answer accuracy.

Contribution

It proposes a dual-modal counterfactual contrastive construction method that enhances egocentric VideoQA by generating and utilizing positive and negative samples for contrastive learning.

Findings

Achieves state-of-the-art results on EgoTaskQA and QAEGO4D datasets.

Effectively models hand-object interactions and multiple events in egocentric videos.

Improves understanding of first-person perspectives in VideoQA tasks.

Abstract

Egocentric Video Question Answering (Egocentric VideoQA) plays an important role in egocentric video understanding, which refers to answering questions based on first-person videos. Although existing methods have made progress through the paradigm of pre-training and fine-tuning, they ignore the unique challenges posed by the first-person perspective, such as understanding multiple events and recognizing hand-object interactions. To deal with these challenges, we propose a Dual-Modal Counterfactual Contrastive Construction (DMC$^3$) framework, which contains an egocentric videoqa baseline, a counterfactual sample construction module and a counterfactual sample-involved contrastive optimization. Specifically, We first develop a counterfactual sample construction module to generate positive and negative samples for textual and visual modalities through event description paraphrasing and core interaction mining, respectively. Then, We feed these samples together with the original samples into the baseline. Finally, in the counterfactual sample-involved contrastive optimization module, we apply contrastive loss to minimize the distance between the original sample features and the positive sample features, while maximizing the distance from the negative samples. Experiments show that our method achieve 52.51\% and 46.04\% on the \textit{normal} and \textit{indirect} splits of EgoTaskQA, and 13.2\% on QAEGO4D, both reaching the state-of-the-art performance.