From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

TL;DR

This paper introduces ARMADA, a scalable and efficient cross-modal knowledge distillation framework that transfers knowledge from large vision-language models, including black-box models, to language-only models without extensive pre-training.

Contribution

ARMADA provides a novel alignment-based method for distilling knowledge from multimodal teachers to language models without modifying the teacher or requiring multimodal pre-training.

Findings

Achieves up to 3.4% improvement on language understanding tasks

Boosts generative reasoning performance by 2.6%

Works effectively with large models like DeBERTa, OPT, and LLaMA

Abstract

Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on language understanding tasks and 2.6% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately.