Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

TL;DR

Taiyi-Diffusion-XL is a bilingual Chinese-English text-to-image model that extends existing models with enhanced vocabulary, position encoding, and vision-language support, achieving superior bilingual image generation and retrieval performance.

Contribution

The paper introduces Taiyi-Diffusion-XL, a novel bilingual text-to-image model with expanded Chinese vocabulary and improved vision-language integration, advancing open-source multilingual multimodal research.

Findings

Bilingual CLIP model outperforms previous models in image-text retrieval.

Taiyi-Diffusion-XL surpasses prior models in bilingual image generation quality.

Open-sourced model fosters further research in multilingual multimodal AI.

Abstract

Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}, fostering further research and collaboration in this domain.