Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

TL;DR

This paper introduces TMKD, a novel knowledge distillation method using dual teachers and multi-view visual priors, significantly improving student model performance by enhancing teacher knowledge quality and semantic alignment.

Contribution

The paper proposes a dual-modality teacher framework with multi-view visual priors and semantic weighting, along with contrastive regularization, to enhance knowledge distillation effectiveness.

Findings

Achieves up to 4.49% performance improvement on benchmarks.

Effectively leverages visual priors and semantic prompts for better knowledge transfer.

Demonstrates robustness across five different benchmark datasets.

Abstract

Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49\%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.