Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching

TL;DR

This paper introduces a flipped knowledge distillation method where large language models learn from smaller, specialized models to improve text matching, addressing architectural differences and enhancing performance in domain-specific tasks.

Contribution

The paper proposes a novel flipped knowledge distillation paradigm using encoder reinterpretation and Margin-aware Contrastive Learning to leverage small models' expertise for large language models.

Findings

Improved performance on financial and healthcare benchmarks.

Effective handling of positive and negative pair similarities.

Successful deployment in real-world online applications.

Abstract

Knowledge distillation typically involves transferring knowledge from a Large Language Model (LLM) to a Smaller Language Model (SLM). However, in tasks such as text matching, fine-tuned smaller models often yield more effective domain-specific representations, as they focus on optimizing the similarity of input pairs. To leverage both the specialized strengths of small models and the rich semantic understanding of LLMs, we introduce a flipped knowledge distillation paradigm, where LLM learns from SLM. Specifically, we address the architectural gap between decoder-only LLMs and smaller encoder-based models by reinterpreting LLMs in an encoder-decoder manner using LoRA. The encoder generates compressed representations, while the decoder maps them to the output space. During training, the encoder produces representations and their similarities, which are then aligned with the similarity scores produced by the teacher, using our proposed Margin-aware Contrastive Learning (MCL) approach. The MCL ensures accurate similarity for both positive and negative pairs, and adaptively handles the internal differences within positive and negative samples. Our paradigm requires only a reasonably good-performing SLM, allowing the LLM to achieve improved performance. Experiments on financial and healthcare benchmarks, as well as real-world applications, confirm its effectiveness, and the model has been fully deployed in an online environment.