Egocentric Activity Recognition and Localization on a 3D Map

TL;DR

This paper introduces a deep probabilistic model that jointly recognizes and localizes egocentric actions within a 3D environment using a hierarchical volumetric representation and video data, advancing scene understanding.

Contribution

The novel model integrates 3D environment context with egocentric video to improve action recognition and localization in known and unknown environments.

Findings

Strong results on action recognition accuracy.

Effective 3D localization in diverse environments.

Generalizes well to unseen scenes.

Abstract

Given a video captured from a first person perspective and the environment context of where the video is recorded, can we recognize what the person is doing and identify where the action occurs in the 3D space? We address this challenging problem of jointly recognizing and localizing actions of a mobile user on a known 3D map from egocentric videos. To this end, we propose a novel deep probabilistic model. Our model takes the inputs of a Hierarchical Volumetric Representation (HVR) of the 3D environment and an egocentric video, infers the 3D action location as a latent variable, and recognizes the action based on the video and contextual cues surrounding its potential locations. To evaluate our model, we conduct extensive experiments on the subset of Ego4D dataset, in which both human naturalistic actions and photo-realistic 3D environment reconstructions are captured. Our method demonstrates strong results on both action recognition and 3D action localization across seen and unseen environments. We believe our work points to an exciting research direction in the intersection of egocentric vision, and 3D scene understanding.