MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts

TL;DR

This paper introduces MultiPL-MoE, a hybrid mixture-of-experts extension for large language models that enhances multilingual code generation by optimizing expert selection at token and segment levels, capturing programming language structures.

Contribution

It proposes a novel MultiPL-MoE architecture combining token and segment-level MoEs with innovative routing and normalization strategies for improved multilingual code generation.

Findings

Effective in improving multilingual code generation performance

Utilizes a novel expert-choice routing strategy

Incorporates a sliding window for segment partitioning

Abstract

Despite LLMs' excellent code creation capabilities, multilingual code generation remains extremely challenging. To address this, we intent to improve the multi-programming-lingual (MultiPL) performance of the base LLMs while retaining the most popular ones using restricted computational resources. We consider MultiPL to be a special case of multiple natural languages and propose a MultiPL extension of LLMs utilizing a hybrid mixture of experts (MoE), called MultiPL-MoE. Specifically, MultiPL-MoE combines two paired MoEs to optimize expert selection at both the token and segment levels. The token-level MoE is a standard upcycling MoE structure with a shared expert and a novel gate weight normalization approach that aids in the final fusion with the segment-level MoE. The segment-level MoE incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages: First, using a sliding window to partition the input token sequence into multiple segments; Then, adopting an expert-choice routing strategy that allows experts to select the top-k segments. The results of the experiment proved the effectiveness of MultiPL-MoE.