Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning

TL;DR

This paper introduces EMBED, a method that transforms exocentric video-language data into egocentric data, enabling improved egocentric video representation learning and achieving state-of-the-art results on multiple benchmarks.

Contribution

The paper presents a novel data transformation framework that adapts exocentric video-language data for egocentric learning, addressing disparities between the two data types.

Findings

Achieved 4.7% improvement on Epic-Kitchens-100 retrieval

Achieved 6.2% improvement on EGTEA classification in zero-shot

Demonstrated strong generalization across various exocentric datasets

Abstract

We present EMBED (Egocentric Models Built with Exocentric Data), a method designed to transform exocentric video-language data for egocentric video representation learning. Large-scale exocentric data covers diverse activities with significant potential for egocentric learning, but inherent disparities between egocentric and exocentric data pose challenges in utilizing one view for the other seamlessly. Egocentric videos predominantly feature close-up hand-object interactions, whereas exocentric videos offer a broader perspective on human activities. Additionally, narratives in egocentric datasets are typically more action-centric and closely linked with the visual content, in contrast to the narrative styles found in exocentric datasets. To address these challenges, we employ a data transformation framework to adapt exocentric data for egocentric training, focusing on identifying specific video clips that emphasize hand-object interactions and transforming narration styles to align with egocentric perspectives. By applying both vision and language style transfer, our framework creates a new egocentric dataset derived from exocentric video-language data. Through extensive evaluations, we demonstrate the effectiveness of EMBED, achieving state-of-the-art results across various egocentric downstream tasks, including an absolute improvement of 4.7% on the Epic-Kitchens-100 multi-instance retrieval and 6.2% on the EGTEA classification benchmarks in zero-shot settings. Furthermore, EMBED enables egocentric video-language models to perform competitively in exocentric tasks. Finally, we showcase EMBED's application across various exocentric datasets, exhibiting strong generalization capabilities when applied to different exocentric datasets.