EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars

TL;DR

This paper introduces EgoAvatar, a novel approach for creating photorealistic, full-body digital avatars from a single egocentric camera, enabling realistic telepresence with accurate motion and appearance modeling.

Contribution

It presents the first unified method to generate and animate full-body avatars from egocentric video, combining geometry, appearance, and motion capture in a person-specific model.

Findings

Outperforms existing baselines in egocentric avatar creation

Achieves accurate full-body motion reconstruction from a single camera

Demonstrates photorealistic avatar rendering in telepresence scenarios

Abstract

Immersive VR telepresence ideally means being able to interact and communicate with digital avatars that are indistinguishable from and precisely reflect the behaviour of their real counterparts. The core technical challenge is two fold: Creating a digital double that faithfully reflects the real human and tracking the real human solely from egocentric sensing devices that are lightweight and have a low energy consumption, e.g. a single RGB camera. Up to date, no unified solution to this problem exists as recent works solely focus on egocentric motion capture, only model the head, or build avatars from multi-view captures. In this work, we, for the first time in literature, propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. We first present a character model that is animatible, i.e. can be solely driven by skeletal motion, while being capable of modeling geometry and appearance. Then, we introduce a personalized egocentric motion capture component, which recovers full-body motion from an egocentric video. Finally, we apply the recovered pose to our character model and perform a test-time mesh refinement such that the geometry faithfully projects onto the egocentric view. To validate our design choices, we propose a new and challenging benchmark, which provides paired egocentric and dense multi-view videos of real humans performing various motions. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods. For more details, code, and data, we refer to our project page.