Translation-Enhanced Multilingual Text-to-Image Generation

TL;DR

This paper explores multilingual text-to-image generation, leveraging neural machine translation and a novel ensemble adapter to improve performance across languages, addressing data scarcity and fairness issues.

Contribution

It systematically studies cross-lingual methods for mTTI and introduces EnsAd, a parameter-efficient approach that enhances multilingual capabilities.

Findings

Translation-enhanced mTTI shows promising results on standard datasets.

EnsAd consistently improves performance across multiple datasets.

The approach mitigates language gaps in multilingual TTI systems.

Abstract

Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated image-caption data in other languages; in the long run, this might widen inequitable access to TTI technology. In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems. We provide two key contributions. 1) Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: Translate Train, Translate Test, and Zero-Shot Transfer. 2) We propose Ensemble Adapter (EnsAd), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance. Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of translation-enhanced mTTI systems and also validate the benefits of the proposed EnsAd which derives consistent gains across all datasets. Further investigations on model variants, ablation studies, and qualitative analyses provide additional insights on the inner workings of the proposed mTTI approaches.