XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts

TL;DR

XFT is a novel training scheme that merges upcycled Mixture-of-Experts with a shared expert mechanism and model merging to significantly enhance instruction-tuned code LLMs, achieving state-of-the-art performance with minimal compute.

Contribution

XFT introduces a simple yet effective method to improve instruction tuning of code LLMs by merging upcycled MoE models with a shared expert mechanism and a learnable merging process.

Findings

Achieves state-of-the-art performance on HumanEval with a tiny 1.3B model.

Improves supervised fine-tuning results by 13% on HumanEval+.

Demonstrates consistent performance gains across multiple benchmarks.

Abstract

We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at https://github.com/ise-uiuc/xft.