MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost

TL;DR

MuLan introduces a lightweight, cost-effective multilingual image generation framework that leverages pre-trained noisy internet image-text pairs, achieving high-quality results across over 110 languages with minimal additional training.

Contribution

The paper presents MuLan, a novel multilingual adapter that enables efficient, high-performance text-to-image generation in hundreds of languages with negligible additional cost.

Findings

Comparable performance across 110+ languages.

CLIP similarity scores nearly match English results.

Seamless integration with existing community tools.

Abstract

In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages.Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.