Test-time Controllable Image Generation by Explicit Spatial Constraint Enforcement

TL;DR

This paper introduces a test-time controllable image generation method that enforces spatial constraints separately, improving layout consistency in generated images without requiring model fine-tuning.

Contribution

It proposes a novel approach to decouple and enforce semantic and geometric spatial constraints during test-time generation, enhancing control and generalizability.

Findings

Achieves 30% relative boost in layout consistency over state-of-the-art methods.

Effectively handles complex spatial conditions with a diffusion-based latents-refill technique.

Demonstrates improved test-time controllability on the Coco-stuff dataset.

Abstract

Recent text-to-image generation favors various forms of spatial conditions, e.g., masks, bounding boxes, and key points. However, the majority of the prior art requires form-specific annotations to fine-tune the original model, leading to poor test-time generalizability. Meanwhile, existing training-free methods work well only with simplified prompts and spatial conditions. In this work, we propose a novel yet generic test-time controllable generation method that aims at natural text prompts and complex conditions. Specifically, we decouple spatial conditions into semantic and geometric conditions and then enforce their consistency during the image-generation process individually. As for the former, we target bridging the gap between the semantic condition and text prompts, as well as the gap between such condition and the attention map from diffusion models. To achieve this, we propose to first complete the prompt w.r.t. semantic condition, and then remove the negative impact of distracting prompt words by measuring their statistics in attention maps as well as distances in word space w.r.t. this condition. To further cope with the complex geometric conditions, we introduce a geometric transform module, in which Region-of-Interests will be identified in attention maps and further used to translate category-wise latents w.r.t. geometric condition. More importantly, we propose a diffusion-based latents-refill method to explicitly remove the impact of latents at the RoI, reducing the artifacts on generated images. Experiments on Coco-stuff dataset showcase 30$\%$ relative boost compared to SOTA training-free methods on layout consistency evaluation metrics.