KD4MT: A Survey of Knowledge Distillation for Machine Translation

TL;DR

This survey reviews the development and application of Knowledge Distillation in Machine Translation, analyzing 105 papers to identify trends, challenges, and future directions in this evolving field.

Contribution

It provides a comprehensive categorization, analysis, and practical guidelines for KD in MT, highlighting research gaps and the impact of large language models.

Findings

Common trends and research gaps identified in KD4MT

Lack of unified evaluation practices in the field

Potential risks like hallucination and bias amplification

Abstract

Knowledge Distillation (KD) as a research area has gained a lot of traction in recent years as a compression tool to address challenges related to ever-larger models in NLP. Remarkably, Machine Translation (MT) offers a much more nuanced take on this narrative: in MT, KD also functions as a general-purpose knowledge transfer mechanism that shapes supervision and translation quality as well as efficiency. This survey synthesizes KD for MT (KD4MT) across 105 papers (through October 1, 2025). We begin by introducing both MT and KD for non-experts, followed by an overview of the standard KD approaches relevant to MT applications. Subsequently, we categorize advances in the KD4MT literature based on (i) their methodological contributions and (ii) their practical applications. Our qualitative and quantitative analyses identify common trends in the field and highlight key research gaps as well as the absence of unified evaluation practice for KD methods in MT. We further provide practical guidelines for selecting a KD method in concrete settings and highlight potential risks associated with the application of KD to MT such as increased hallucination and bias amplification. Finally, we discuss the role of LLMs in re-shaping the KD4MT field. To support further research, we complement our survey with a publicly available database summarizing the main characteristics of the surveyed KD methods and a glossary of key terms.