Masking Teacher and Reinforcing Student for Distilling Vision-Language Models

TL;DR

This paper introduces Masters, a novel framework for distilling vision-language models by masking and progressively restoring the teacher's weights, combined with offline reinforcement learning to improve student learning stability and performance.

Contribution

The paper proposes a mask-progressive RL distillation method that enhances knowledge transfer from large teachers to small students in vision-language models.

Findings

Improved student model performance on vision-language tasks.

Stable and efficient knowledge distillation process.

Effective offline RL rewards for guiding student learning.

Abstract

Large-scale vision-language models (VLMs) have recently achieved remarkable multimodal understanding, but their massive size makes them impractical for deployment on mobile or edge devices. This raises the need for compact yet capable VLMs that can efficiently learn from powerful large teachers. However, distilling knowledge from a large teacher to a small student remains challenging due to their large size gap: the student often fails to reproduce the teacher's complex, high-dimensional representations, leading to unstable learning and degraded performance. To address this, we propose Masters (Masking Teacher and Reinforcing Student), a mask-progressive reinforcement learning (RL) distillation framework. Masters first masks non-dominant weights of the teacher to reduce unnecessary complexity, then progressively restores the teacher by gradually increasing its capacity during training. This strategy allows the student to learn richer representations from the teacher in a smooth and stable manner. To further refine knowledge transfer, Masters integrates an offline RL stage with two complementary rewards: an accuracy reward that measures the correctness of the generated responses, and a distillation reward that quantifies the ease of transferring responses from teacher to student. Unlike online think-answer RL paradigms that are computationally expensive and generate lengthy responses, our offline RL leverages pre-generated responses from masked teachers. These provide rich yet efficient guidance, enabling students to achieve strong performance without requiring the think-answer process.