# You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions

## TL;DR

This paper introduces a learning-based method to estimate a wearable camera wearer's 3D body pose by analyzing interactions with others in egocentric videos, leveraging observable partner poses to infer the out-of-view body pose.

## Contribution

It presents a novel approach that uses the observable body poses of interaction partners to infer the wearer's pose, improving egocentric pose estimation accuracy.

## Key findings

- Enhanced pose estimation accuracy over previous methods
- Effective modeling of interaction dynamics for pose inference
- Demonstrated on diverse dyadic interaction datasets

## Abstract

The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera. We propose a learning-based approach to estimate the camera wearer's 3D body pose from egocentric video sequences. Our key insight is to leverage interactions with another person---whose body pose we can directly observe---as a signal inherently linked to the body pose of the first-person subject. We show that since interactions between individuals often induce a well-ordered series of back-and-forth responses, it is possible to learn a temporal model of the interlinked poses even though one party is largely out of view. We demonstrate our idea on a variety of domains with dyadic interaction and show the substantial impact on egocentric body pose estimation, which improves the state of the art. Video results are available at http://vision.cs.utexas.edu/projects/you2me/