uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data

TL;DR

uCLIP introduces a lightweight, parameter-efficient method for extending vision-language models to low-resource languages without requiring additional image-text data, significantly improving multilingual retrieval performance.

Contribution

The paper presents a novel, minimal-training framework that aligns multilingual representations using a small projection module and English anchors, enhancing performance in underrepresented languages.

Findings

Significant improvements in retrieval accuracy for five underrepresented languages.

Effective multilingual alignment achieved without additional image-text or text-text pairs.

The approach is robust and data-efficient, requiring only 1.7M parameters to train.

Abstract

Contrastive Language-Image Pre-training (CLIP) has demonstrated strong generalization across a wide range of visual tasks by leveraging large-scale English-image pairs. However, its extension to low-resource languages remains limited due to the scarcity of high-quality multilingual image-text data. Existing multilingual vision-language models exhibit consistently low retrieval performance in underrepresented languages including Czech, Finnish, Croatian, Hungarian, and Romanian on the Crossmodal-3600 (XM3600) benchmark. To address this, we propose a lightweight and data-efficient framework for multilingual vision-language alignment. Our approach requires no image-text pairs or text-text pairs and freezes both the pretrained image encoder and multilingual text encoder during training. Only a compact 1.7M-parameter projection module is trained, using a contrastive loss over English representations as semantic anchors. This minimal training setup enables robust multilingual alignment even for languages with limited supervision. Extensive evaluation across multiple multilingual retrieval benchmarks confirms the effectiveness of our method, showing significant gains in five underrepresented languages where existing models typically underperform. These findings highlight the effectiveness of our pivot-based, parameter-efficient alignment strategy for inclusive multimodal learning.