Mix-of-Language-Experts Architecture for Multilingual Programming

TL;DR

This paper presents MoLE, a novel architecture that efficiently balances language-specific specialization and parameter sharing for multilingual programming tasks using a mix-of-experts approach with LoRA modules.

Contribution

MoLE introduces a joint optimization of shared and language-specific LoRA modules, enabling efficient and specialized multilingual programming models.

Findings

MoLE outperforms single shared LLMs in accuracy across multiple programming languages.

MoLE achieves greater parameter efficiency than training separate language-specific models.

MoLE effectively balances specialization and efficiency in multilingual programming tasks.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in aiding developers with tasks like code comprehension, generation, and translation. Supporting multilingual programming -- i.e., coding tasks across multiple programming languages -- typically requires either (1) finetuning a single LLM across all programming languages, which is cost-efficient but sacrifices language-specific specialization and performance, or (2) finetuning separate LLMs for each programming language, which allows for specialization but is computationally expensive and storage-intensive due to the duplication of parameters. This paper introduces MoLE (Mix-of-Language-Experts), a novel architecture that balances efficiency and specialization for multilingual programming. MoLE is composed of a base model, a shared LoRA (low-rank adaptation) module, and a collection of language-specific LoRA modules. These modules are jointly optimized during the finetuning process, enabling effective knowledge sharing and specialization across programming languages. During inference, MoLE automatically routes to the language-specific LoRA module corresponding to the programming language of the code token being generated. Our experiments demonstrate that MoLE achieves greater parameter efficiency compared to training separate language-specific LoRAs, while outperforming a single shared LLM finetuned for all programming languages in terms of accuracy.