Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

TL;DR

This paper demonstrates that combining sparse Mixture-of-Experts with instruction tuning significantly enhances large language models' performance, outperforming dense models while using fewer computational resources.

Contribution

It shows that instruction tuning greatly benefits MoE models, leading to superior performance on benchmarks with reduced FLOPs, challenging traditional design principles.

Findings

MoE models underperform dense models without instruction tuning

Instruction tuning boosts MoE models' performance substantially

FLAN-MOE-32B surpasses FLAN-PALM-62B on benchmarks with fewer FLOPs

Abstract

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.