EAGLE: Egocentric AGgregated Language-video Engine

TL;DR

EAGLE introduces a unified egocentric video understanding framework with a large-scale dataset and a multimodal large language model, enabling comprehensive analysis of human activities and intentions from first-person videos.

Contribution

The paper presents EAGLE, a novel integrated model and the first large-scale instruction-tuning dataset for egocentric videos, advancing holistic understanding across multiple tasks.

Findings

EAGLE outperforms existing models in various egocentric video tasks.

The EAGLE-400K dataset enhances model training and generalization.

Proposed evaluation metrics facilitate comprehensive assessment.

Abstract

The rapid evolution of egocentric video analysis brings new insights into understanding human activities and intentions from a first-person perspective. Despite this progress, the fragmentation in tasks like action recognition, procedure learning, and moment retrieval, \etc, coupled with inconsistent annotations and isolated model development, hinders a holistic interpretation of video content. In response, we introduce the EAGLE (Egocentric AGgregated Language-video Engine) model and the EAGLE-400K dataset to provide a unified framework that integrates various egocentric video understanding tasks. EAGLE-400K, the \textit{first} large-scale instruction-tuning dataset tailored for egocentric video, features 400K diverse samples to enhance a broad spectrum of tasks from activity recognition to procedure knowledge learning. Moreover, EAGLE, a strong video multimodal large language model (MLLM), is designed to effectively capture both spatial and temporal information. In addition, we propose a set of evaluation metrics designed to facilitate a thorough assessment of MLLM for egocentric video understanding. Our extensive experiments demonstrate EAGLE's superior performance over existing models, highlighting its ability to balance task-specific understanding with holistic video interpretation. With EAGLE, we aim to pave the way for research opportunities and practical applications in real-world scenarios.