L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts

TL;DR

L-MoE introduces an end-to-end trainable framework combining Mixture of Experts and Low-Rank Adaptation, enabling efficient, modular, and specialized language models through dynamic expert composition.

Contribution

The paper proposes L-MoE, a novel approach that unifies MoE and LoRA into a differentiable, end-to-end trainable model with task-specific low-rank experts and dynamic routing.

Findings

Achieves parameter-efficient language modeling with dynamic expert composition.

Allows end-to-end training of a modular MoE architecture.

Demonstrates improved scalability and specialization in language models.

Abstract

The Mixture of Experts (MoE) architecture enables the scaling of Large Language Models (LLMs) to trillions of parameters by activating a sparse subset of weights for each input, maintaining constant computational cost during inference. Concurrently, Low-Rank Adaptation (LoRA) has emerged as a dominant technique for parameter-efficiently fine-tuning LLMs on specialized tasks. In this work, we unify these two paradigms into a novel, end-to-end trainable framework named L-MoE: a Lightweight Mixture of LoRA Experts. L-MoE redefines MoE experts not as dense feed-forward networks, but as a collection of task-specialized, low-rank adapters. A lightweight gating network, trained jointly with the experts, learns to dynamically compose these LoRA adapters by computing a weighted average of their parameters for each input token. This composition is fully differentiable, allowing gradients from a standard auto-regressive language modeling objective to flow back through the entire architecture, simultaneously refining both the expert adapters and the routing strategy. This approach creates a highly parameter-efficient MoE model that is modular by design, allows for dynamic skill composition, and is trainable from end-to-end. We present the formal mathematical framework for L-MoE, detailing the differentiable routing mechanism and the joint optimization objective, thereby providing a new path toward building more efficient, scalable, and specialized language models.