Text2Place: Affordance-aware Text Guided Human Placement

TL;DR

This paper introduces Text2Place, a novel method for realistic human placement in scenes guided by text, which learns semantic masks and performs subject-conditioned inpainting without large-scale training, enabling diverse and realistic scene compositions.

Contribution

The work presents the first effective solution for realistic human placement in diverse scenes using text-guided semantic masks and inpainting, eliminating the need for extensive training.

Findings

Achieves highly realistic scene compositions preserving background and subject identity.

Outperforms strong baselines in realistic human placement tasks.

Enables downstream applications like scene hallucination and attribute editing.

Abstract

For a given scene, humans can easily reason for the locations and pose to place objects. Designing a computational model to reason about these affordances poses a significant challenge, mirroring the intuitive reasoning abilities of humans. This work tackles the problem of realistic human insertion in a given background scene termed as \textbf{Semantic Human Placement}. This task is extremely challenging given the diverse backgrounds, scale, and pose of the generated person and, finally, the identity preservation of the person. We divide the problem into the following two stages \textbf{i)} learning \textit{semantic masks} using text guidance for localizing regions in the image to place humans and \textbf{ii)} subject-conditioned inpainting to place a given subject adhering to the scene affordance within the \textit{semantic masks}. For learning semantic masks, we leverage rich object-scene priors learned from the text-to-image generative models and optimize a novel parameterization of the semantic mask, eliminating the need for large-scale training. To the best of our knowledge, we are the first ones to provide an effective solution for realistic human placements in diverse real-world scenes. The proposed method can generate highly realistic scene compositions while preserving the background and subject identity. Further, we present results for several downstream tasks - scene hallucination from a single or multiple generated persons and text-based attribute editing. With extensive comparisons against strong baselines, we show the superiority of our method in realistic human placement.