Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos

TL;DR

This paper introduces a method to separate human hand and environment in egocentric videos, improving robotic task performance by using a diffusion model for better scene inpainting and representation.

Contribution

The work presents a novel factorization approach and a diffusion-based inpainting model that enhances scene understanding and downstream robotic applications from egocentric videos.

Findings

VIDM improves inpainting quality in egocentric videos

Factored representation aids object detection and 3D reconstruction

Enhances learning of reward functions and policies from videos

Abstract

The analysis and use of egocentric videos for robotic tasks is made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide a valuable signal, e.g. the hand pose may suggest what kind of object is being held. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos.