LACONIC: A 3D Layout Adapter for Controllable Image Creation

TL;DR

LACONIC introduces a 3D-aware adapter for pretrained diffusion models, enabling controllable, scene-consistent image synthesis with camera and geometry controls, while supporting scene editing and generalization.

Contribution

We propose a novel 3D-aware conditioning method and adapter network that enhances pretrained text-to-image models with scene and object-level 3D control.

Findings

Supports camera and explicit 3D geometry conditioning

Enables scene editing like object positioning and resizing

Shows strong generalization with lightweight design

Abstract

Existing generative approaches for guided image synthesis of multi-object scenes typically rely on 2D controls in the image or text space. As a result, these methods struggle to maintain and respect consistent three-dimensional geometric structure, underlying the scene. In this paper, we propose a novel conditioning approach, training method and adapter network that can be plugged into pretrained text-to-image diffusion models. Our approach provides a way to endow such models with 3D-awareness, while leveraging their rich prior knowledge. Our method supports camera control, conditioning on explicit 3D geometries and, for the first time, accounts for the entire context of a scene, i.e., both on and off-screen items, to synthesize plausible and semantically rich images. Despite its multi-modal nature, our model is lightweight, requires a reasonable number of data for supervised learning and shows remarkable generalization power. We also introduce methods for intuitive and consistent image editing and restyling, e.g., by positioning, rotating or resizing individual objects in a scene. Our method integrates well within various image creation workflows and enables a richer set of applications compared to previous approaches.