Efficient Sub-structured Knowledge Distillation

TL;DR

This paper introduces an efficient knowledge distillation method for structured prediction models that matches sub-structure predictions, avoiding complex decoding and enabling faster training.

Contribution

It proposes a simple, efficient approach that transfers knowledge by locally matching sub-structures, reducing training time and improving performance.

Findings

Outperforms previous methods on structured prediction tasks.

Halves the training time per epoch.

Encourages better internal mimicry of teacher models.

Abstract

Structured prediction models aim at solving a type of problem where the output is a complex structure, rather than a single variable. Performing knowledge distillation for such models is not trivial due to their exponentially large output space. In this work, we propose an approach that is much simpler in its formulation and far more efficient for training than existing approaches. Specifically, we transfer the knowledge from a teacher model to its student model by locally matching their predictions on all sub-structures, instead of the whole output space. In this manner, we avoid adopting some time-consuming techniques like dynamic programming (DP) for decoding output structures, which permits parallel computation and makes the training process even faster in practice. Besides, it encourages the student model to better mimic the internal behavior of the teacher model. Experiments on two structured prediction tasks demonstrate that our approach outperforms previous methods and halves the time cost for one training epoch.