MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

TL;DR

This paper introduces MELLA, a dataset designed to improve low-resource multilingual models by enhancing both linguistic skills and cultural awareness through targeted data collection and fine-tuning.

Contribution

The study presents MELLA, a novel multimodal, multilingual dataset that boosts low-resource language performance by integrating cultural groundedness and linguistic capability.

Findings

Performance improved across eight languages after fine-tuning on MELLA.

Models produce more detailed 'thick descriptions' with cultural context.

Enhancement is due to both cultural knowledge and linguistic capability gains.

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable performance in high-resource languages. However, their effectiveness diminishes significantly in the contexts of low-resource languages. Current multilingual enhancement methods are often limited to text modality or rely solely on machine translation. While such approaches help models acquire basic linguistic capabilities and produce "thin descriptions", they neglect the importance of multimodal informativeness and cultural groundedness, both of which are crucial for serving low-resource language users effectively. To bridge this gap, in this study, we identify two significant objectives for a truly effective MLLM in low-resource language settings, namely 1) linguistic capability and 2) cultural groundedness, placing special emphasis on cultural awareness. To achieve these dual objectives, we propose a dual-source strategy that guides the collection of data tailored to each goal, sourcing native web alt-text for culture and MLLM-generated captions for linguistics. As a concrete implementation, we introduce MELLA, a multimodal, multilingual dataset. Experiment results show that after fine-tuning on MELLA, there is a general performance improvement for the eight languages on various MLLM backbones, with models producing "thick descriptions". We verify that the performance gains are from both cultural knowledge enhancement and linguistic capability enhancement. Our dataset can be found at https://opendatalab.com/applyMultilingualCorpus.