Canvas3D: Empowering Precise Spatial Control for Image Generation with Constraints from a 3D Virtual Canvas

TL;DR

Canvas3D introduces a 3D virtual canvas system that allows users to precisely control spatial arrangements in image generation, improving accuracy and interactivity over existing methods.

Contribution

The paper presents Canvas3D, a novel interactive system that uses a 3D engine to enable precise spatial control in image generation from text descriptions.

Findings

Canvas3D outperforms baseline in spatial control accuracy.

Users report higher satisfaction with Canvas3D's interactivity.

System demonstrates improved overall user experience.

Abstract

Generative AI (GenAI) has significantly advanced the ease and flexibility of image creation. However, it remains a challenge to precisely control spatial compositions, including object arrangement and scene conditions. To bridge this gap, we propose Canvas3D, an interactive system leveraging a 3D engine to enable precise spatial manipulation for image generation. Upon user prompt, Canvas3D automatically converts textual descriptions into interactive objects within a 3D engine-driven virtual canvas, empowering direct and precise spatial configuration. These user-defined arrangements generate explicit spatial constraints that guide generative models in accurately reflecting user intentions in the resulting images. We conducted a closed-end comparative study between Canvas3D and a baseline system. And an open-ended study to evaluate our system "in the wild". The result indicates that Canvas3D outperforms the baseline on spatial control, interactivity, and overall user experience.