cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillation

TL;DR

This paper introduces cViL, a method for training vision-language models in new languages using knowledge distillation from English models, achieving improved accuracy in Japanese and Hindi visual question answering tasks.

Contribution

The paper presents a novel cross-lingual training pipeline that leverages English models and knowledge distillation to efficiently extend vision-language models to other languages.

Findings

Outperforms state-of-the-art by 4.4% in Japanese VQA accuracy.

Outperforms state-of-the-art by 13.4% in Hindi VQA accuracy.

Provides a large-scale VQA dataset in Japanese and Hindi.

Abstract

Vision-and-language tasks are gaining popularity in the research community, but the focus is still mainly on English. We propose a pipeline that utilizes English-only vision-language models to train a monolingual model for a target language. We propose to extend OSCAR+, a model which leverages object tags as anchor points for learning image-text alignments, to train on visual question answering datasets in different languages. We propose a novel approach to knowledge distillation to train the model in other languages using parallel sentences. Compared to other models that use the target language in the pretraining corpora, we can leverage an existing English model to transfer the knowledge to the target language using significantly lesser resources. We also release a large-scale visual question answering dataset in Japanese and Hindi language. Though we restrict our work to visual question answering, our model can be extended to any sequence-level classification task, and it can be extended to other languages as well. This paper focuses on two languages for the visual question answering task - Japanese and Hindi. Our pipeline outperforms the current state-of-the-art models by a relative increase of 4.4% and 13.4% respectively in accuracy.