Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models

TL;DR

This paper introduces PLAST, a method for enhancing multilingual capabilities of large vision-language models by fine-tuning language-specific shallow layers, achieving significant improvements with minimal parameter updates.

Contribution

The paper presents a novel approach to identify and fine-tune language-specific layers in LVLMs, improving multilingual understanding efficiently.

Findings

PLAST improves multilingual performance on MM-Bench and MMMB.

Only 14% of parameters need tuning for effective enhancement.

PLAST generalizes well to low-resource and complex visual reasoning tasks.

Abstract

Large vision-language models (LVLMs) have demonstrated exceptional capabilities in understanding visual information with human languages but also exhibit an imbalance in multilingual capabilities. In this work, we delve into the multilingual working pattern of LVLMs and identify a salient correlation between the multilingual understanding ability of LVLMs and language-specific neuron activations in shallow layers. Building on this insight, we introduce PLAST, a training recipe that achieves efficient multilingual enhancement for LVLMs by Precise LAnguage-Specific layers fine-Tuning. PLAST first identifies layers involved in multilingual understanding by monitoring language-specific neuron activations. These layers are then precisely fine-tuned with question-translation pairs to achieve multilingual alignment. Our empirical results on MM-Bench and MMMB demonstrate that PLAST effectively improves the multilingual capabilities of LVLMs and achieves significant efficiency with only 14% of the parameters tuned. Further analysis reveals that PLAST can be generalized to low-resource and complex visual reasoning tasks, facilitating the language-specific visual information engagement in shallow layers.