EPIC Fields: Marrying 3D Geometry and Video Understanding

TL;DR

EPIC Fields is a new dataset augmenting EPIC-KITCHENS with 3D camera info, enabling research in neural rendering and video understanding without complex camera reconstruction, and includes benchmarks for dynamic object segmentation.

Contribution

It introduces EPIC Fields, a dataset that combines 3D camera data with egocentric videos, facilitating neural rendering research and addressing photogrammetry challenges.

Findings

EPIC Fields reconstructs 96% of videos from EPIC-KITCHENS.

The dataset includes 19 million frames across 45 kitchens.

Benchmark results highlight current limitations and potential of neural rendering.

Abstract

Neural rendering is fuelling a unification of learning, 3D geometry and video understanding that has been waiting for more than two decades. Progress, however, is still hampered by a lack of suitable datasets and benchmarks. To address this gap, we introduce EPIC Fields, an augmentation of EPIC-KITCHENS with 3D camera information. Like other datasets for neural rendering, EPIC Fields removes the complex and expensive step of reconstructing cameras using photogrammetry, and allows researchers to focus on modelling problems. We illustrate the challenge of photogrammetry in egocentric videos of dynamic actions and propose innovations to address them. Compared to other neural rendering datasets, EPIC Fields is better tailored to video understanding because it is paired with labelled action segments and the recent VISOR segment annotations. To further motivate the community, we also evaluate two benchmark tasks in neural rendering and segmenting dynamic objects, with strong baselines that showcase what is not possible today. We also highlight the advantage of geometry in semi-supervised video object segmentations on the VISOR annotations. EPIC Fields reconstructs 96% of videos in EPICKITCHENS, registering 19M frames in 99 hours recorded in 45 kitchens.