Generative Blocks World: Moving Things Around in Pictures

TL;DR

This paper introduces Generative Blocks World, a method for editing generated images by manipulating 3D primitives, enabling accurate, consistent, and flexible scene modifications with improved visual quality.

Contribution

It presents a novel scene representation using convex 3D primitives and a flow-based image generation conditioned on depth and texture hints, enhancing editability and visual fidelity.

Findings

Outperforms prior methods in visual fidelity

Enables accurate object and camera movements

Preserves object identity during edits

Abstract

We describe Generative Blocks World to interact with the scene of a generated image by manipulating simple geometric abstractions. Our method represents scenes as assemblies of convex 3D primitives, and the same scene can be represented by different numbers of primitives, allowing an editor to move either whole structures or small details. Once the scene geometry has been edited, the image is generated by a flow-based method, which is conditioned on depth and a texture hint. Our texture hint takes into account the modified 3D primitives, exceeding the texture-consistency provided by existing techniques. These texture hints (a) allow accurate object and camera moves and (b) preserve the identity of objects. Our experiments demonstrate that our approach outperforms prior works in visual fidelity, editability, and compositional generalization.