A Practical Investigation of Spatially-Controlled Image Generation with Transformers

TL;DR

This paper systematically compares transformer-based models for spatially-controlled image generation, proposing a simple baseline, exploring sampling enhancements, and clarifying the role of adapter-based methods to guide future research.

Contribution

It provides a comprehensive, controlled experimental analysis of transformer-based spatially-controlled image generation, introducing a baseline and clarifying the effectiveness of various techniques.

Findings

Control token prefilling is an effective baseline.

Extending classifier-free guidance improves control consistency.

Adapter-based approaches help mitigate forgetting with limited data.

Abstract

Enabling image generation models to be spatially controlled is an important area of research, empowering users to better generate images according to their own fine-grained specifications via e.g. edge maps, poses. Although this task has seen impressive improvements in recent times, a focus on rapidly producing stronger models has come at the cost of detailed and fair scientific comparison. Differing training data, model architectures and generation paradigms make it difficult to disentangle the factors contributing to performance. Meanwhile, the motivations and nuances of certain approaches become lost in the literature. In this work, we aim to provide clear takeaways across generation paradigms for practitioners wishing to develop transformer-based systems for spatially-controlled generation, clarifying the literature and addressing knowledge gaps. We perform controlled experiments on ImageNet across diffusion-based/flow-based and autoregressive (AR) models. First, we establish control token prefilling as a simple, general and performant baseline approach for transformers. We then investigate previously underexplored sampling time enhancements, showing that extending classifier-free guidance to control, as well as softmax truncation, have a strong impact on control-generation consistency. Finally, we re-clarify the motivation of adapter-based approaches, demonstrating that they mitigate "forgetting" and maintain generation quality when trained on limited downstream data, but underperform full training in terms of generation-control consistency.