Rescaling Egocentric Vision

TL;DR

This paper presents EPIC-KITCHENS-100, a significantly expanded egocentric video dataset with dense annotations, enabling new research challenges like action detection, recognition, and generalisation over time.

Contribution

The paper introduces a novel pipeline for annotating egocentric videos, resulting in a larger, more detailed dataset that supports multiple new research challenges.

Findings

EPIC-KITCHENS-100 contains 100 hours of video with 20 million frames.

Annotations are 54% denser and 128% more complete than previous versions.

The dataset supports six new challenges including action detection and cross-modal retrieval.

Abstract

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the "test of time" - i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics