A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

TL;DR

This paper introduces DC-CLIP, a lightweight multilingual vision-language model trained through a two-stage distillation and alignment process, achieving high performance in English and Chinese with reduced resource requirements.

Contribution

It presents a simple, effective framework for multilingual CLIP compression, enabling deployment on resource-constrained devices with competitive accuracy.

Findings

DC-CLIP outperforms existing models in zero-shot image classification.

The model achieves high accuracy with less training data.

The training framework effectively aligns visual and multilingual textual features.

Abstract

Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.