MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis

TL;DR

MARS is a novel auto-regressive framework for text-to-image synthesis that integrates pre-trained language models with visual understanding, achieving high-quality, bilingual, and efficient image generation.

Contribution

Introduces SemVIE, a new component that combines language and visual processing in auto-regressive models, enhancing T2I synthesis with multi-stage training and bilingual capabilities.

Findings

Achieves high-quality T2I generation with only 9% of the GPU days of SD1.5.

Supports bilingual (English and Chinese) image and text generation.

Significantly improves text-image alignment and image detail granularity.

Abstract

Auto-regressive models have made significant progress in the realm of language generation, yet they do not perform on par with diffusion models in the domain of image synthesis. In this work, we introduce MARS, a novel framework for T2I generation that incorporates a specially designed Semantic Vision-Language Integration Expert (SemVIE). This innovative component integrates pre-trained LLMs by independently processing linguistic and visual information, freezing the textual component while fine-tuning the visual component. This methodology preserves the NLP capabilities of LLMs while imbuing them with exceptional visual understanding. Building upon the powerful base of the pre-trained Qwen-7B, MARS stands out with its bilingual generative capabilities corresponding to both English and Chinese language prompts and the capacity for joint image and text generation. The flexibility of this framework lends itself to migration towards any-to-any task adaptability. Furthermore, MARS employs a multi-stage training strategy that first establishes robust image-text alignment through complementary bidirectional tasks and subsequently concentrates on refining the T2I generation process, significantly augmenting text-image synchrony and the granularity of image details. Notably, MARS requires only 9% of the GPU days needed by SD1.5, yet it achieves remarkable results across a variety of benchmarks, illustrating the training efficiency and the potential for swift deployment in various applications.