When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA

TL;DR

This paper investigates the effectiveness of knowledge distillation for CLIP-style vision-language models in VQA, revealing that stronger teachers do not always produce better students and highlighting challenges in scaling existing frameworks.

Contribution

It provides the first systematic analysis of KD across various CLIP models for VQA, challenging assumptions about teacher strength and model scaling.

Findings

Stronger teachers do not always improve student performance.

Existing distillation methods often degrade performance when scaled.

Challenges in applying KD to large-scale multimodal models are identified.

Abstract

Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to small-scale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models.