AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models

TL;DR

AMMKD introduces an adaptive multi-modal multi-teacher distillation framework that effectively reduces model size and complexity for vision-language retrieval tasks, maintaining high performance on benchmarks.

Contribution

The paper presents a novel framework combining multi-modal feature fusion, multi-teacher distillation, and adaptive optimization for lightweight vision-language models.

Findings

Achieves superior retrieval performance with reduced model complexity.

Effectively aligns teacher-student outputs using KL scatter.

Demonstrates flexibility and effectiveness across multiple datasets.

Abstract

The success of large-scale visual language pretraining (VLP) models has driven widespread adoption of image-text retrieval tasks. However, their deployment on mobile devices remains limited due to large model sizes and computational complexity. We propose Adaptive Multi-Modal Multi-Teacher Knowledge Distillation (AMMKD), a novel framework that integrates multi-modal feature fusion, multi-teacher distillation, and adaptive optimization to deliver lightweight yet effective retrieval models. Specifically, our method begins with a feature fusion network that extracts and merges discriminative features from both the image and text modalities. To reduce model parameters and further improve performance, we design a multi-teacher knowledge distillation framework to pre-train two CLIP teacher models. We decouple modalities by pre-computing and storing text features as class vectors via the teacher text encoder to enhance efficiency. To better align teacher and student outputs, we apply KL scatter for probability distribution matching. Finally, we design an adaptive dynamic weighting scheme that treats multi-teacher distillation as a multi-objective optimization problem. By leveraging gradient space diversity, we dynamically adjust the influence of each teacher, reducing conflicts and guiding the student toward more optimal learning directions. Extensive experiments on three benchmark datasets demonstrate that AMMKD achieves superior performance while significantly reducing model complexity, validating its effectiveness and flexibility.

Tables

Table 1. Table 1: Methods Performance Comparison

Methods	ImageNet-tiny		Flower102		UCF-101		Caltech101		DTD		OxfordPets		EuroSAT
	Resnet	Vit	Resnet	Vit	Resnet	Vit	Resnet	Vit	Resnet	Vit	Resnet	Vit	Resnet	Vit
Cumstomkd	69.01	69.01	58.00	61.72	72.71	81.01	80.16	68.21	33.25	38.92	49	34.67	94.91	93.19
FitNet	68.95	68.17	59.10	61.72	73.16	80.16	80.55	68.25	33.51	38.83	49.17	38.58	94.87	94.39
RKD	69.08	66.06	62.21	71.98	80.04	86.89	81.67	77.92	31.12	42.55	47.42	37.50	91.46	93.98
SP	52.46	49.15	32.48	41.27	72.86	41.06	74.89	38.37	19.77	38.56	47.83	37.50	75.98	87.02
FNKD	66.46	64.09	55.68	61.78	69.94	79.19	76.08	68.08	29.88	39.01	48.08	37.42	95.43	94.83
KD	60.25	61.85	58.55	61.29	72.64	72.02	79.99	78.04	32.80	33.28	48.67	49.06	94.56	94.70
DKD	62.12	55.78	52.35	45.12	69.01	65.43	72.46	67.89	29.57	25.98	47.68	43.21	93.03	93.57
Logit	60.29	58.79	49.88	56.47	67.23	68.60	70.62	68.46	27.31	27.95	47.50	44.47	95.26	94.30
GA	68.19	65.85	58.12	62.45	71.60	72.80	79.99	76.49	32.54	35.49	49.92	50.50	94.67	94.72
Vlite	68.76	67.00	56.84	62.64	65.94	63.42	72.03	71.94	27.57	32.79	47.00	47.64	95.39	94.82
DHO	61.81	59.06	57.63	56.71	70.27	67.53	76.47	74.94	29.70	33.81	46.33	44.55	92.98	93.96
AMMKD(OURS)	69.53	70.25	74.56	73.63	86.45	87.93	91.71	90.26	46.22	46.63	46.71	43.92	94.57	94.11

Table 2. Table 2: Performance on Flower102 and UCF-101, where “MT” denotes “multi-teacher” and “DWS” represents multi-teacher dynamic weight selection.

	$L_{CLIP}$	$L_{KL}$	$L_{MSE}$	$L{R}_{Cosine}$	$L{R}_{Fixed}$	Data Aug	Flower102	UCF-101
CLIPlarge (Radford et al. 2021)	–	–	–	–	–	–	69.96	66.38
DFN (Fang et al. 2023)	–	–	–	–	–	–	69.73	64.38
Base	$✓$	–	–	–	$✓$	–	60.46	76.03
	$✓$	–	–	$✓$	–	–	60.83	78.42
	$✓$	–	–	$✓$	–	$✓$	61.78	61.78
Multi-Teacher	$✓$	$✓$	–	–	$✓$	–	65.28	76.47
	$✓$	–	$✓$	–	$✓$	–	66.34	78.84
	$✓$	$✓$	$✓$	–	$✓$	–	68.45	80.46
	$✓$	$✓$	$✓$	$✓$	–	–	70.56	83.56
	$✓$	$✓$	$✓$	$✓$	–	$✓$	72.42	85.42
MT + DWS	$✓$	$✓$	$✓$	$✓$	–	$✓$	73.63	87.93

Table 3. Table 3: Proportion of different loss functions.

Proportion	ImageNet-tiny	Flower102	UCF-101
0.5:1:1	67.77	69.45	83.64
1:0.5:1	65.75	70.87	80.94
1:1:0.5	68.48	72.59	82.78
1:1:1	70.25	73.63	87.93

Table 4. Table 4: Multi-Teacher Dynamic Selection Mechanism.

DWSM	ImageNet-tiny	Flower102	UCF-101
Base	59.45	61.78	76.93
Avg	60.11	68.12	86.32
LSR	68.15	72.20	84.78
DSW	70.25	73.63	87.93

Table 5. Table 5: Different Numbers of Teacher Models.

Numbers	ImageNet-tiny	Flower102	UCF-101
1	67.45	69.84	86.47
2	70.25	73.63	87.93
3	69.45	70.46	86.41
4	66.57	73.95	85.46

Table 6. Table 6: Different Sizes of Student Models.

Sizes	ImageNet-tiny	Flower102	UCF-101
6layer+64	67.05	66.73	83.13
8layer+128	69.52	70.59	84.03
6layer+512	70.25	73.63	87.93

Equations

$$p(P|Q)=\frac{exp(u_i\cdot w_i/\tau )}{{\sum }_{j=1}^{N}exp(u_i\cdot w_j/\tau )}.$$

$$p(Q|P)=\frac{exp(w_i\cdot u_i/\tau )}{{\sum }_{j=1}^{N}exp(w_i\cdot u_j/\tau )},$$

$$p_k^T(P|Q)=\frac{exp(u_i^T\cdot w_i^T/{\tau }^T)}{{\sum }_{j=1}^{N}exp(u_i^T\cdot w_j^T/{\tau }^T)},$$

$$p^S(P|Q)=\frac{exp(u_i^S\cdot w_i^S/{\tau }^S)}{{\sum }_{j=1}^{N}exp(u_i^S\cdot w_j^S/{\tau }^S)}.$$

$$p^S(P|Q)=\frac{exp(u_i^S\cdot w_i^S/{\tau }^S)}{{\sum }_{j=1}^{N}exp(u_i^S\cdot w_j^S/{\tau }^S)},$$

$$p^S(P|Q)=\frac{exp(u_i^S\cdot w_i^S/{\tau }^S)}{{\sum }_{j=1}^{N}exp(u_i^S\cdot w_j^S/{\tau }^S)}.$$

$$L_{kl}(P|Q)=KL(p_k^T(P|Q),p^S(P|Q)),$$

$$L_{kl}(Q|P)=KL(p_k^T(Q|P),p^S(Q|P)),$$

$$L_{clip}=CE\left(y,\cos \left(u_i,w_i\right)\right),$$

$$L_{mse}=MSE(u_i^T,u_i^S)+MSE(w_i^T,w_i^S).$$

$$L=L_{kl}+L_{clip}+L_{mse},$$

$$L_{kl}={\lambda }_1{L}_{kl}(P|Q)+{\lambda }_2{L}_{kl}(Q|P),$$

$${\lambda }_1=\frac{r_1}{r_1+r_2},$$

$${\lambda }_2=\frac{r_2}{r_1+r_2}.$$

$$\underset{\alpha }{\min }$$

$$\sum _{k=1}^K{\alpha }_k=1,{\alpha }_k\ge 0,\forall k\in [1:K],$$

Full text

Introduction

Multimodal modeling focuses on learning how to represent and summarize multimodal data in a way that exploits the complementarity and redundancy of multiple modalities. The heterogeneity of multimodal data makes it challenging to construct such representations. For example, language is often symbolic, while audio and visual forms will be represented as signals. Unimodal representations are responsible for representing information as numerical vectors that computers can process or further abstract into higher-level feature vectors, whereas multimodal representations refer to learning better feature representations by taking advantage of complementarities between multiple modalities and eliminating redundancies between modalities.

CLIP (Radford et al. 2021) has taken a significant step forward in the multimodal domain, with the main contribution being the use of unsupervised textual information as a supervised signal to learn visual features. This training is performed via the contrast loss method to achieve uniform embedding space learning of multimodal signals, thus efficiently aligning the representations of different modalities. This approach not only improves the performance of the model in multimodal tasks but also provides important directions for future research. Due to the significant gaps between different modalities, multimodal data not only provides effective information, but also brings higher information redundancy. However, most existing fusion methods focus mainly on simply integrating multimodal features or employing a single optimization strategy during joint training, which usually leads to unsatisfactory performance.

Knowledge distillation (KD) (Xu et al. 2024) can be categorized into single-teacher and multi-teacher paradigms. The single-teacher setting transfers knowledge from a single model to a student, but the teacher’s capacity inherently limits the student’s performance. In contrast, multi-teacher distillation aggregates knowledge from multiple teachers, providing richer supervision and improved generalization.

However, vanilla multi-teacher distillation often treats each teacher equally or fuses their outputs naively, without considering their expertise, modality relevance, or potential conflicts. This may lead to ineffective or even contradictory guidance for the student model, limiting its performance. Recent studies (Zhang, Chen, and Wang 2023) propose adaptive strategies to address these challenges by weighting or selecting teacher knowledge more intelligently.

In the practical operation of the distillation process, knowledge transfer is usually achieved through a simple method of averaging the knowledge or information extracted from each collection of teacher models. However, this approach fails to adequately take into account differences in diversity across teacher models within the set. As a result, competing and interfering noises may be generated during the distillation process, thus affecting the overall effect.

Facing these challenges, we propose an adaptive multimodal multi-teacher knowledge distillation framework, as shown in Figure 2. This can better alleviate the problem of large parameter scales of multimodal models. It can keep the student model as close as possible to the teacher model in terms of performance, while drastically reducing the number of parameters of the model, to satisfy the needs in graphic retrieval, graphic QA, pure text comprehension, and pure visual comprehension tasks. To further improve the accuracy of the distilled student model, it is crucial to take full advantage of the integration of the multi-teacher model while minimizing the possible adverse effects among the teacher models. Therefore, in the framework of multi-teacher distillation, each teacher model should be provided with a weight to help the student model learn and absorb knowledge from different teachers more comprehensively.

To summarize, our main contributions are as follows:

1. We propose a multimodal multi-teacher knowledge distillation framework that leverages guidance from multiple teachers to improve the cross-modal retrieval performance of a lightweight CLIP model.

2. We introduce a novel loss function designed to align the text and image modalities of the teacher model with those of the student model, respectively, thereby enhancing modality-specific knowledge transfer.

3. To address the potential presence of noisy or weak-performing teachers, we present a multi-teacher dynamic weighting strategy. This strategy employs a dual optimization algorithm to weigh and extract useful knowledge from all available teachers adaptively.

4. Extensive experiments on several public multimodal datasets demonstrate that our AMMKD significantly outperforms traditional single-teacher distillation baseline approaches.

Related Work

Multi-teacher Knowledge Distillation

Within the framework of multi-teacher knowledge distillation, each teacher network offers soft-label predictions, which are then combined to direct the student network’s learning. AVER (Wang et al. 2022a) adopts a straightforward approach by assigning equal weight to each teacher’s prediction, treating all contributions as equally valuable. In contrast, RLKD (Yuan et al. 2021) introduces reinforcement learning to improve the selection process, filtering out less suitable teachers before averaging the remaining predictions. This refinement aims to reduce the influence of teachers whose guidance may be sub-optimal for the student. Other methods focus on tailoring the weight each teacher contributes based on their unique qualities. For example, EBKD (Kwon et al. 2020) employs information entropy to assign distinct weights to each teacher, adapting based on the diversity of predictions. Similarly, CA-MKD (Zhang, Chen, and Wang 2022) evaluates the reliability of individual teacher predictions by calculating cross-entropy with ground-truth labels, ensuring that teachers with higher predictive confidence have a greater impact on the student’s learning. On the other hand, AEKD (Du et al. 2020) takes an entirely different perspective by examining teacher diversity within the gradient space, which helps to understand how variations among teachers contribute to the student’s training process. Some multi-teacher KD techniques maintain fixed or uniform weights for simplicity (You et al. 2017; Fukuda et al. 2017; Wu, Chiu, and Wu 2019), while Yuang et al.(Liu, Zhang, and Wang 2020) propose to leverage latent factors to calculate dynamic weights, adjusting each teacher’s influence according to underlying factors in the data. Despite their benefits, these label-independent weighting strategies present some limitations. For example, entropy-based weighting (Kwon et al. 2020) can inadvertently prioritize predictions with low variance, which may not always be reliable. Optimization-based strategies, on the other hand, may rely too heavily on majority consensus, making them susceptible to noisy data that could mislead the student (Du et al. 2020). With a meta-weight network, Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning (MMKD) (Zhang, Chen, and Wang 2023) proposes to adaptively integrate knowledge from multiple teachers and enhance student model performance by considering both teacher diversity and knowledge compatibility.

Multi-modal Knowledge Distillation

Multi-Modal Knowledge Distillation (MKD) has emerged as a powerful technique aimed at improving knowledge transfer across various modalities, including images (Li et al. 2023, 2021; Yang et al. 2022), text (Jiao et al. 2019; Chen et al. 2021; Zheng et al. 2024), and audio (Schmid, Koutini, and Widmer 2023; Choi et al. 2022). By integrating information from these diverse sources, MKD leverages the complementary strengths of each modality, thereby enhancing the performance of student models in tasks that necessitate a multi-faceted understanding and representation. Wang et al.(Wang et al. 2022b) enhance the generalization capabilities of models on multi-modal tasks by adapting distillation strategies that exploit the complementarity of different modalities, which improves the performance of individual encoders through cross-modal knowledge transfer. Additionally, Chen et al.(Liang et al. 2024) introduce Module-wise Adaptive Distillation, explicitly designed for multimodal foundation models, allowing for knowledge distillation on a module-level basis. Aiming at achieving an alignment of the feature maps and attention distributions between the teacher and student models, DistillVLM (Fang et al. 2021) yields a more compact yet effective visual-linguistic compression technique. CLIP (Radford et al. 2021) demonstrates that models can effectively learn to associate images with text descriptions without the need for explicit pairings. By utilizing natural language as a supervisory signal, CLIP produces a robust zero-shot image classification model that generalizes across a wide array of visual concepts. Building on this foundation, TinyCLIP (Wu et al. 2023) aims to mimic the affinity between teacher and student models while inheriting weights to develop a compact yet practical student model. Furthermore, CLIP-KD (Yang et al. 2024) explores strategies for distilling large CLIP models into smaller, more efficient student models, focusing on techniques such as feature mimicry and interactive contrastive learning to bolster the student model’s generalization capabilities and performance on zero-shot tasks.

Methodology

Preliminary

The general framework of the method in this paper is shown in Figure 2 and contains two pre-trained multimodal teacher models and one multimodal student model. We input $M$ paired image and text datasets $X=\{(P_{i},Q_{i})\}_{i=1}^{M}$, where $p_{i}\in\mathbb{R}^{C\times H\times W}$ denotes the $i$th image sample with channel $C$, width $W$ and height $H$, and ${{Q}_{i}}$ is the $i$th text sample. The model in our framework utilizes an image feature extraction module and a text feature extraction module to extract rich information from the image and text samples, respectively. Then, image features and text features are mapped into a vector space for comparative learning using a projector to obtain the Image-to-Text probability distribution and the Text-to-Image probability distribution, respectively. This network is trained in two steps. First, the teacher network is trained using the cross-entropy loss function to obtain the CLIP teacher model. Then, the parameters of the pre-trained teachers are frozen, and the multimodal knowledge imparted by the two teachers is used to guide the training of the multimodal students. The key elements are described in detail in the following subsections.

Multi-Modal Feature Extraction

In this paper, CLIP is used as a base model.CLIP consists of two encoders, one for images and the other for text. Given a labeled visual recognition dataset $X=\{(P_{i},Q_{i})\}_{i=1}^{M}$, which includes a set of $N$ class names $c={{\{{{c}_{j}}\}}_{j=1}^{N}}$, CLIP uses a template to generate a description ${{Q}_{i}}$ as “a photo of a $\left\{{{{c}_{j}}}\right\}$”.Each text description ${{Q}_{i}}$ is then input to a text encoder ${{f}_{T}}$ to obtain normalized text features ${{w}_{i}}=\frac{{{f}_{T}}({{Q}_{i}})}{||{{f}_{T}}({{Q}_{i}})|{{|}_{2}}}\in{{R}^{d}}$ , where $d$ denotes the feature dimension. Similarly, an image ${{P}_{i}}$ is input to the image encoder ${{f}_{I}}$ which takes it as input and generates normalized image features ${{u}_{i}}=\frac{{{f}_{I}}({{P}_{i}})}{{{||{{f}_{I}}({{P}_{i}})||}_{2}}}\in{{R}^{d}}$.The image-to-text output probability is calculated as follows:

[TABLE]

CLIP conducts a symmetric image-text alignment. Given the text embedding as the anchor, the text-to-image output probability is formulated as:

[TABLE]

where $\tau$ is the temperature parameter.

Multi-Teacher Multi-Modal Knowledge Distillation

After identifying the multimodal student network, we propose a novel multimodal multi-teacher knowledge distillation to comprehensively aggregate the knowledge of multiple pre-trained multimodal teachers, thereby effectively training the multimodal student to achieve better performance. The overall loss function of the student consists of distillation losses (i.e., ${{L}_{kl}}$ and ${{L}_{mse}}$) and student loss (i.e., ${{L}_{clip}}$). The teacher and student image-to-text contrastive distributions ${{p}_{k}^{T}}(P|Q)$and ${{p}^{S}}(P|Q)$ are formulated as:

[TABLE]

[TABLE]

Similarly, text-to-image contrastive distributions ${{p}_{k}^{T}}(P|Q)$and ${{p}^{S}}(P|Q)$ as follows:

[TABLE]

[TABLE]

This paper aligns the contrast distribution between teacher and student by KL divergence loss. For image to text and text to image, the distillation loss for the $k$ teacher and the student loss ${{L}_{clip}}$ can be calculated as follows:

[TABLE]

[TABLE]

[TABLE]

where $KL$ (·) is the Kullback-Leibler divergence function [25]. $CE$ (·) represents the contrast learning loss function, and $y$ represents the label. $cos$(·) represents the cosine similarity calculation.

Align feature vectors between teachers and students to directly reduce knowledge gaps. Intuitively, if student characteristics could be perfectly aligned with teacher characteristics, their performance gaps would disappear. This work uses Mean Squared Error (MSE) to guide students to imitate the teacher’s image and text feature vectors:

[TABLE]

By combining Equations (7), (8), (9), and (10), we can get the overall loss function of the student model as follows:

[TABLE]

[TABLE]

where ${\lambda}_{1}$ and ${\lambda}_{2}$ are the dynamic parameters that balance the influence of teacher loss, and the combination strategy of the teacher model can be flexibly optimized. In essence, this parameter-guided method introduces an intelligent regulation mechanism in the collaborative training of multi-teacher models, so that the weight allocation of teacher models can be continuously optimized with the deepening of the training process to meet the learning needs of student models. The details are explained in the next section.

Multi-Teacher Dynamic Weight Selection

As shown in Equation (12), if the losses are added with a parameter of 1, all teacher losses are considered equally. However, since teachers may offer different learning directions to students, conflicts and competition may arise among teachers. Therefore, the final direction of learning will be determined by the primary teacher, thus weakening the guidance of other teachers. To this end, we propose a multi-teacher dynamic selection mechanism, which can be regarded as a dynamic weighting strategy that can adaptively extract knowledge from all teachers.

First, the similarity between each teacher image and the current label text is taken as a dynamic weight, denoted as ${r}_{1}$ and ${r}_{2}$, as shown in the module of Figure 2 DWS. The resulting formula is shown in Equation (12), where:

[TABLE]

[TABLE]

Second, when faced with internal incoherence in multi-teacher knowledge distillation (KD), we choose to re-imagine multi-teacher knowledge distillation as a multi-objective optimization (MOO) framework from the perspective of gradient space. This change aims to consider the knowledge contribution of multiple teacher models simultaneously, in order to find a student model configuration that can harmoniously integrate the advantages of each teacher. Next, we will elaborate on how to use the multi-gradient descent algorithm (MGDA) to find the optimal solution in this complex multi-objective environment.

On the basis of MGDA, it is assumed that for each task, KKT (Kuarush-Huhn-Tucker, a necessary condition for optimal solution of nonlinear programming) holds:

There is ${{\alpha}_{1}},\dots,{{\alpha}_{K}}\geq 0$, s.t. $\sum_{k=1}^{K}{{{\alpha}_{k}}}=1$,$\sum_{k=1}^{K}{{{\alpha}_{k}}}{{\nabla}_{p}}CE\left({{{p}^{S}},{{p}_{k}^{T}}}\right)=0$. In the shared layer, there is no feasible direction of decline; that is, it belongs to the global best, and the optimal solution for all teachers is found.

For any teacher $k$, ${{\nabla}_{p}}CE\left({{{p}^{S}},{{p}_{k}^{T}}}\right)=0$.

Any solution satisfying the above conditions is a Pareto stationary point, and every optimization solution is a stationary point, and vice versa is not true, so we rewrite the optimization problem as follows:

[TABLE]

where $g(P|Q)=CE(p^{S}(P|Q),p^{T}(P|Q))$and$g(Q|P)=CE(p^{S}(Q|P),p^{T}(Q|P))$ Represent the image-to-text cross entropy and the text-to-image cross entropy of the student and teacher models, respectively, and are gradient steps. In this way, the optimization direction will be less affected by those deviating or noisy teachers, thus reducing performance degradation. Our approach can be viewed as a dynamic weighting strategy for each teacher to refine knowledge during the adaptive training of the student network.

Experiments

In this section, we first describe the datasets and experimental setup. We then evaluate the performance of the proposed method on the collected multimodal datasets and provide a detailed analysis to demonstrate its effectiveness.

Experimental Settings

Datasets

We conducted experiments on multiple image classification datasets, including ImageNet-tiny (Deng et al. 2009), Flower102(Duman and Süzen 2022), UCF-101(Soomro, Zamir, and Shah 2012), Caltech101(Bansal et al. 2023), DTD(Cimpoi et al. 2014), Oxford Pet(Parkhi et al. 2012), and EuroSAT(Helber et al. 2019). More details about the dataset descriptions are provided in the supplementary materials.

Models

We choose widely used ResNet-18 (He et al. 2016) and ViT-B/16 (Dosovitskiy et al. 2020) to perform image classification, because they are representative architectures for convolutional neural network and vision Transformer.

Training details

We employ a variety of data enhancement techniques, covering fundamental transformations (e.g., flip, rotate) and advanced methods (e.g., Cutout (DeVries and Taylor 2017), Mixup (Zhang et al. 2017), etc.), aiming to improve the model generalization capability. All methods in this paper are implemented using the PyTorch (Paszke et al. 2017) framework and trained on NVIDIA A100 GPUs (40GB). We chose OpenAI-clip and DFN2B-CLIP as teacher networks for images and text. Adam, with a learning rate of 0.0001, was used as the optimizer in all experiments. In the experiments, the batch size was set to 64. The dropout regularization probability was set to be 0.5 to reduce overfitting. The temperatures $\tau 1$, $\tau 2$, and $\tau 3$ were set to be 4.

Results and Analysis

The AMMKD proposed in this paper demonstrates significant advantages on both ResNet and ViT, the two mainstream architectures, proving its powerful knowledge fusion and transfer capabilities.

The experimental data of the ViT architecture further verified the universality of the method (1 and Figure 1). In the ImageNet-tiny benchmark test, AMMKD led all comparison methods with an accuracy of 69.53 (Cumstomkd ranked second with 69.01). On the fine-grained recognition task Flower102, its performance of 74.56 shows a significant advantage (FitNet only 59.1). It is worth noting that AMMKD’s significant lead in Caltech101 (91.71 vs RKD 81.67) and DTD texture classification tasks (46.22 vs KD 32.8) highlights its cross-modal alignment ability for complex visual features. Although the gap with Cumstomkd on the OxfordPets dataset was slight (46.71 vs 49.00), it still ranked in the first tier with a high score of 94.57 in the EuroSAT remote sensing scenario. The radar chart of the ResNet architecture is provided in the supplementary material.

These results indicate that AMMKD effectively resolves the information redundancy and conflict issues of traditional knowledge distillation in multi-teacher scenarios, providing a new paradigm for multimodal knowledge transfer.

Ablation Studies

According to the comprehensive analysis of the data in Table 2, the MT + DWS model achieved the highest accuracy rates of 73.63$\%$ and 87.93$\%$ respectively on the Flower102 and UCF-101 datasets, significantly outperforming the CLIPlarge benchmark model (with a 21.55$\%$ improvement on UCF-101). The multi-teacher framework (MT) has an average improvement of over 6$\%$ compared to the Base model (for example, UCF-101 has risen from 78.84$\%$ to 85.42$\%$). Its performance has been further optimized after the introduction of the DWS weighting mechanism (UCF-101 has improved by another 2.51$\%$), proving that it effectively enhances the time series modeling ability. Data augmentation has a remarkable effect on the Flower102 small sample dataset (with a 5.88$\%$ improvement in the Base model), but the gain for video tasks is limited. In the training strategy, the fixed learning rate ($LR_{Fixed}$) is more stable than the cosine strategy, and the MSE loss only brings a marginal improvement. Overall, it is indicated that image and video classification benefits more from the collaborative optimization of multi-teacher distillation and DWS.

Proportion of different loss functions

Table 3 Experiments prove that the weight ratio of the loss function significantly affects the model performance. The 1:1:1 equilibrium strategy achieves the optimum on ImageNet-tiny (70.25$\%$), Flower102 (73.63$\%$), and UCF-101 (87.93$\%$). During weight adjustment, reducing any loss function (such as 0.5:1:1 or 1:1:0.5) leads to performance degradation, especially for the video dataset UCF-101, which is the most sensitive (with a maximum decline of 7$\%$). The image classification task Flower102 is relatively robust in weight fine-tuning (fluctuation 4$\%$), but the balanced weight still leads steadily. Video tasks strictly rely on a 1:1:1 ratio to ensure multi-loss collaboration. The conclusion emphasizes that cross-scenario models must strictly maintain the balance of loss function weights to avoid performance degradation caused by local weakening.

Multi-Teacher Dynamic Selection Mechanism

In this experiment, the performance of four methods, namely Base, Avg, ratio of similarity to label (LSR), and Dynamic similarity Weight (DSW), was compared on three datasets, namely ImageNet-tiny, Flower102, and UCF-101. The Table 4 shows that the DSW method demonstrated a comprehensive leading advantage, achieving the highest accuracy rates on ImageNet-tiny (70.25$\%$), Flower102 (73.63$\%$), and UCF-101 (87.93$\%$). Particularly on the small dataset ImageNet-tiny, it improved by nearly 11$\%$ compared to the Base method. Highlight its strong adaptability to small-scale data. The performance of the LSR method varies. Although it outperforms the Avg method in ImageNet-tiny (68.15$\%$) and Flower102 (72.20$\%$), it is surpassed by Avg (86.32$\%$) on the video dataset UCF-101 (84.78$\%$), revealing the limitations of its temporal feature capture. The Base method, as the benchmark, ranked at the bottom among all tasks, with a gap of 11$\%$ compared to the optimal DSW on UCF-101, exposing the shortcomings of traditional models. The conclusion shows that DSW, with its stable improvement across datasets (an average increase of 9.2 to 12.5 percentage points compared to Base), has become the solution with the best generalization ability, and is particularly suitable for resource-constrained small sample scenarios and complex video recognition tasks.

Different Numbers of Teacher Models

Table 5 Experimental results show that: The dual-teacher model combination (open-clip+DFN-clip) achieved optimal or near-optimal performance on the three datasets of ImageNet-tiny, Flower102, and UCF-101, with a maximum improvement of 3.8 percentage points (Flower102). It indicates that the complementarity of heterogeneous teachers can effectively enhance the effect of knowledge distillation. However, expanding the number of teacher models instead led to performance fluctuations: adding a third model, MetaCLIP, caused a general decline (such as a 3.17-point drop in Flower102), indicating that the new model introduced knowledge redundancy or conflicts; When the fourth openAI-clip model was added, Flower102 saw a slight rebound to 73.95 due to domain adaptability, but the rest of the datasets deteriorated significantly (ImageNet-tiny plunged by 3.68 points), reflecting the risk of task dependence in multi-teacher integration. Overall, the combination of the two models achieves the best balance in generalization and compatibility. For subsequent expansions, it is necessary to strictly evaluate the compatibility of the new model with core teachers and the specific task requirements to avoid overfitting and negative transfer effects.

Different Sizes of Student Models

According to the experimental data analysis in Table 6, the expansion of the student model scale (including the increase in the number of Transformer layers and feature dimensions of the visual and text encoders) can significantly improve the performance of cross-modal tasks, specifically manifested as: When the feature dimension rose from 64 to 512 (6-layer model), Flower102 increased by nearly seven percentage points and UCF-101 increased by more than 4.8 percentage points. It is particularly worth noting that under similar model scales, the 6-layer +512 structure is comprehensively superior to the 8-layer +128 structure (for example, Flower102 improves by 3.03$\%$, UCF-101 improves by 3.9$\%$), proving that increasing the feature dimension is more effective than simply stacking the number of layers. Meanwhile, it was found that the improvement in model scale for complex tasks (such as Flower102/UCF-101) (approximately 7$\%$) was much higher than that for the simple task ImageNet-tiny (only 3$\%$). In conclusion, expanding the model scale, primarily by increasing the feature dimensions, can significantly enhance cross-modal capabilities, and the effect is more prominent in complex visual tasks, providing key guidance for efficient model design.

Conclusion

In this paper, we propose AMMKD, an adaptive multimodal multi-teacher knowledge distillation method. AMMKD introduces a novel multi-teacher distillation framework capable of effectively transferring multimodal knowledge, thereby guiding the training of student models toward improved performance. Furthermore, we designed a dynamic teacher selection mechanism that adaptively adjusts the importance weights of different teachers, enhancing both modality adaptation and classification accuracy of the student model.

Extensive experiments conducted on several publicly available multimodal datasets demonstrate that AMMKD significantly outperforms traditional single-teacher distillation approaches. By integrating information from multiple modalities and capturing data features more comprehensively, our method also enhances the generalization capability of the student model.

We will further explore multimodal multi-teacher distillation, including the integration of emerging modalities such as remote sensing, medical images, and sensor data.