Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment

TL;DR

This paper introduces a multi-teacher guided distillation approach that enhances the efficiency and performance of small language models, making deployment faster and more resource-efficient without sacrificing understanding or generation quality.

Contribution

It proposes a novel collaborative distillation strategy with multiple teachers, integrating their outputs and features to improve student model performance and generalization.

Findings

Student models achieve lower perplexity and higher quality in text generation.

The method outperforms existing distillation approaches in multiple evaluation metrics.

Enhanced semantic understanding and task adaptability demonstrated across various NLP tasks.

Abstract

This paper addresses the challenges of high computational cost and slow inference in deploying large language models. It proposes a distillation strategy guided by multiple teacher models. The method constructs several teacher models and integrates their output probability distributions and intermediate semantic features. This guides the student model to learn from multiple sources of knowledge. As a result, the student model gains stronger language understanding and generation ability while maintaining a small parameter size. To achieve this, the paper introduces a weighted output fusion mechanism, a feature alignment loss function, and an entropy-driven dynamic teacher weighting strategy. These components improve the quality and stability of knowledge transfer during distillation. Under multi-teacher guidance, the student model captures semantic information more effectively and demonstrates strong performance across multiple evaluation metrics. In particular, the method shows high consistency in expression, generalization ability, and task adaptability in tasks such as language modeling, text generation, and multi-task learning. The experiments compare the proposed method with several widely adopted distillation approaches. The results further confirm its overall advantages in perplexity, distillation loss, and generation quality. This study provides a feasible technical path for the efficient compression of large-scale language models. It also demonstrates the effectiveness of multi-teacher collaborative mechanisms in complex language modeling tasks.