The One Where They Reconstructed 3D Humans and Environments in TV Shows

TL;DR

This paper introduces an automated method to reconstruct 3D models of humans and environments from TV shows by leveraging the repetition of scenes, enabling enhanced analysis and applications like re-identification and editing.

Contribution

The authors propose a novel approach that reconstructs 3D environments and human poses from entire TV show seasons, improving downstream tasks and applications.

Findings

Successfully reconstructed 3D environments and human models from seven TV shows.

Enhanced 3D human pose and position recovery using environment context.

Demonstrated applications in re-identification, gaze estimation, and image editing.

Abstract

TV shows depict a wide variety of human behaviors and have been studied extensively for their potential to be a rich source of data for many applications. However, the majority of the existing work focuses on 2D recognition tasks. In this paper, we make the observation that there is a certain persistence in TV shows, i.e., repetition of the environments and the humans, which makes possible the 3D reconstruction of this content. Building on this insight, we propose an automatic approach that operates on an entire season of a TV show and aggregates information in 3D; we build a 3D model of the environment, compute camera information, static 3D scene structure and body scale information. Then, we demonstrate how this information acts as rich 3D context that can guide and improve the recovery of 3D human pose and position in these environments. Moreover, we show that reasoning about humans and their environment in 3D enables a broad range of downstream applications: re-identification, gaze estimation, cinematography and image editing. We apply our approach on environments from seven iconic TV shows and perform an extensive evaluation of the proposed system.