Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding

TL;DR

This paper introduces R-MSD, a multi-sample distillation framework for video understanding that reduces variance and improves knowledge transfer by leveraging a teacher pool and quality-aware filtering, outperforming single-sample methods.

Contribution

The paper presents R-MSD, a novel multi-sample distillation approach that models teacher sampling variance and uses a task-adaptive teacher pool for more reliable supervision in video understanding.

Findings

R-MSD outperforms single sample distillation across multiple benchmarks.

Significant improvements on VideoMME, Video-MMMU, and MathVerse datasets.

Marginal gains from baseline SFT+RL 4B model under same training budget.

Abstract

Traditional black-box distillation for Large Vision-Language Models (LVLMs) typically relies on a single teacher response per input, which often yields high-variance responses and format inconsistencies in multimodal or temporal scenarios. To mitigate this unreliable supervision, we propose R-MSD (Reliable Multi-Sample Distillation), a framework that explicitly models teacher sampling variance to enhance distillation stability. Rather than relying on a single teacher response, our approach leverages a task-adaptive teacher pool to provide robust supervision tailored to both closed-ended and open-ended reasoning. By integrating quality-aware signal matching with an adversarial distillation objective, our approach effectively filters teacher noise while maximizing knowledge transfer. Extensive evaluations across comprehensive video understanding benchmarks demonstrate that R-MSD consistently outperforms single sample distillation methods. We additionally include an original SFT+RL 4B baseline under the same training budget, which shows only marginal gains, while our method achieves significant improvements. With a 4B student model, our approach delivers gains on VideoMME (+1.5%), Video-MMMU (+3.2%), and MathVerse (+3.6%).