Hallucinating Pose-Compatible Scenes
Tim Brooks, Alexei A. Efros

TL;DR
This paper introduces a large-scale generative model that creates human pose-compatible scenes from pose inputs, revealing insights into scene understanding and enabling various applications like scene hallucination and pose animation.
Contribution
The authors develop a scale-augmented GAN with a novel pose conditioning mechanism trained on a massive dataset, advancing pose-conditioned scene generation capabilities.
Findings
Model outperforms baselines in human placement accuracy
Generates diverse and high-quality scene samples
Enables multiple applications like scene hallucination and pose animation
Abstract
What does human pose tell us about a scene? We propose a task to answer this question: given human pose as input, hallucinate a compatible scene. Subtle cues captured by human pose -- action semantics, environment affordances, object interactions -- provide surprising insight into which scenes are compatible. We present a large-scale generative adversarial network for pose-conditioned scene generation. We significantly scale the size and complexity of training data, curating a massive meta-dataset containing over 19 million frames of humans in everyday environments. We double the capacity of our model with respect to StyleGAN2 to handle such complex data, and design a pose conditioning mechanism that drives our model to learn the nuanced relationship between pose and scene. We leverage our trained model for various applications: hallucinating pose-compatible scene(s) with or without…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
Methods*Communicated@Fast*How Do I Communicate to Expedia? · PatchGAN · Concatenated Skip Connection · Convolution · Batch Normalization · Weight Demodulation · HuMan(Expedia)||How do I get a human at Expedia? · Sigmoid Activation · Dropout · Pix2Pix
