Heterogeneous-Branch Collaborative Learning for Dialogue Generation

TL;DR

This paper introduces a novel heterogeneous-branch collaborative learning approach for dialogue generation that enhances branch diversity through attribute-based training and dual group distillation, leading to improved performance.

Contribution

It proposes a dual group-based knowledge distillation method that increases branch heterogeneity by considering dialogue attributes, addressing the homogeneity problem in collaborative learning.

Findings

Outperforms state-of-the-art collaborative learning methods

Significantly improves branch heterogeneity

Achieves better dialogue generation quality

Abstract

With the development of deep learning, advanced dialogue generation methods usually require a greater amount of computational resources. One promising approach to obtaining a high-performance and lightweight model is knowledge distillation, which relies heavily on the pre-trained powerful teacher. Collaborative learning, also known as online knowledge distillation, is an effective way to conduct one-stage group distillation in the absence of a well-trained large teacher model. However, previous work has a severe branch homogeneity problem due to the same training objective and the independent identical training sets. To alleviate this problem, we consider the dialogue attributes in the training of network branches. Each branch learns the attribute-related features based on the selected subset. Furthermore, we propose a dual group-based knowledge distillation method, consisting of positive distillation and negative distillation, to further diversify the features of different branches in a steadily and interpretable way. The proposed approach significantly improves branch heterogeneity and outperforms state-of-the-art collaborative learning methods on two widely used open-domain dialogue datasets.