Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?

TL;DR

This paper evaluates whether egocentric video-language models truly understand hand-object interactions, introduces a new benchmark EgoHOIBench to test their limitations, and proposes EgoNCE++ to improve their fine-grained understanding and performance.

Contribution

It introduces EgoHOIBench for assessing egocentric models' understanding of hand-object interactions and proposes EgoNCE++ to enhance model performance through improved supervision and contrastive learning.

Findings

EgoVLMs struggle with simple modifications in interaction descriptions.

EgoNCE++ significantly improves model performance on various tasks.

Models show greater difficulty recognizing verbs than nouns in interactions.

Abstract

Egocentric video-language pretraining is a crucial step in advancing the understanding of hand-object interactions in first-person scenarios. Despite successes on existing testbeds, we find that current EgoVLMs can be easily misled by simple modifications, such as changing the verbs or nouns in interaction descriptions, with models struggling to distinguish between these changes. This raises the question: Do EgoVLMs truly understand hand-object interactions? To address this question, we introduce a benchmark called EgoHOIBench, revealing the performance limitation of current egocentric models when confronted with such challenges. We attribute this performance gap to insufficient fine-grained supervision and the greater difficulty EgoVLMs experience in recognizing verbs compared to nouns. To tackle these issues, we propose a novel asymmetric contrastive objective named EgoNCE++. For the video-to-text objective, we enhance text supervision by generating negative captions using large language models or leveraging pretrained vocabulary for HOI-related word substitutions. For the text-to-video objective, we focus on preserving an object-centric feature space that clusters video representations based on shared nouns. Extensive experiments demonstrate that EgoNCE++ significantly enhances EgoHOI understanding, leading to improved performance across various EgoVLMs in tasks such as multi-instance retrieval, action recognition, and temporal understanding. Our code is available at https://github.com/xuboshen/EgoNCEpp.