VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis

TL;DR

VersaGen is a versatile AI model that provides users with flexible visual controls for text-to-image synthesis, allowing detailed scene and subject manipulation beyond existing methods.

Contribution

It introduces a novel approach that supports multiple types of visual controls in T2I synthesis, enhancing creative flexibility and user experience.

Findings

Effective control over individual subjects and scenes

Improved image quality with optimization strategies

Validated versatility through experiments on COCO and Sketchy

Abstract

Despite the rapid advancements in text-to-image (T2I) synthesis, enabling precise visual control remains a significant challenge. Existing works attempted to incorporate multi-facet controls (text and sketch), aiming to enhance the creative control over generated images. However, our pilot study reveals that the expressive power of humans far surpasses the capabilities of current methods. Users desire a more versatile approach that can accommodate their diverse creative intents, ranging from controlling individual subjects to manipulating the entire scene composition. We present VersaGen, a generative AI agent that enables versatile visual control in T2I synthesis. VersaGen admits four types of visual controls: i) single visual subject; ii) multiple visual subjects; iii) scene background; iv) any combination of the three above or merely no control at all. We train an adaptor upon a frozen T2I model to accommodate the visual information into the text-dominated diffusion process. We introduce three optimization strategies during the inference phase of VersaGen to improve generation results and enhance user experience. Comprehensive experiments on COCO and Sketchy validate the effectiveness and flexibility of VersaGen, as evidenced by both qualitative and quantitative results.