EgoAnimate: Generating Human Animations from Egocentric top-down Views

TL;DR

EgoAnimate introduces a novel generative approach based on Stable Diffusion to reconstruct realistic frontal human avatars from egocentric top-down views, enabling more accessible telepresence with minimal input.

Contribution

This is the first method to use a generative backbone for reconstructing animatable avatars from egocentric inputs, reducing training complexity and enhancing generalizability.

Findings

Successfully generates realistic frontal views from occluded top-down images.

Enables avatar motion generation from a single egocentric image.

Improves telepresence system accessibility and versatility.

Abstract

An ideal digital telepresence experience requires accurate replication of a person's body, clothing, and movements. To capture and transfer these movements into virtual reality, the egocentric (first-person) perspective can be adopted, which enables the use of a portable and cost-effective device without front-view cameras. However, this viewpoint introduces challenges such as occlusions and distorted body proportions. There are few works reconstructing human appearance from egocentric views, and none use a generative prior-based approach. Some methods create avatars from a single egocentric image during inference, but still rely on multi-view datasets during training. To our knowledge, this is the first study using a generative backbone to reconstruct animatable avatars from egocentric inputs. Based on Stable Diffusion, our method reduces training burden and improves generalizability. Inspired by methods such as SiTH and MagicMan, which perform 360-degree reconstruction from a frontal image, we introduce a pipeline that generates realistic frontal views from occluded top-down images using ControlNet and a Stable Diffusion backbone. Our goal is to convert a single top-down egocentric image into a realistic frontal representation and feed it into an image-to-motion model. This enables generation of avatar motions from minimal input, paving the way for more accessible and generalizable telepresence systems.