Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers

TL;DR

This paper introduces transfer capability distillation to improve MoE Transformers' downstream performance by leveraging vanilla models as teachers, addressing transfer capability issues and enhancing transfer learning effectiveness.

Contribution

It proposes a novel transfer capability distillation method that guides MoE models using vanilla models to boost downstream task performance.

Findings

MoE models underperform vanilla models in transfer capability

Transfer capability distillation significantly improves MoE downstream performance

Experimental results on BERT show notable performance gains

Abstract

Recently, Mixture of Experts (MoE) Transformers have garnered increasing attention due to their advantages in model capacity and computational efficiency. However, studies have indicated that MoE Transformers underperform vanilla Transformers in many downstream tasks, significantly diminishing the practical value of MoE models. To explain this issue, we propose that the pre-training performance and transfer capability of a model are joint determinants of its downstream task performance. MoE models, in comparison to vanilla models, have poorer transfer capability, leading to their subpar performance in downstream tasks. To address this issue, we introduce the concept of transfer capability distillation, positing that although vanilla models have weaker performance, they are effective teachers of transfer capability. The MoE models guided by vanilla models can achieve both strong pre-training performance and transfer capability, ultimately enhancing their performance in downstream tasks. We design a specific distillation method and conduct experiments on the BERT architecture. Experimental results show a significant improvement in downstream performance of MoE models, and many further evidences also strongly support the concept of transfer capability distillation. Finally, we attempt to interpret transfer capability distillation and provide some insights from the perspective of model feature.