MoLoRA: Composable Specialization via Per-Token Adapter Routing

TL;DR

MoLoRA introduces per-token adapter routing for multimodal and mixed-capability models, enabling modular, specialized, and efficient inference by dynamically selecting adapters per token, outperforming larger models in reasoning tasks.

Contribution

The paper proposes per-token routing and MoLoRA for composable specialization, allowing dynamic adapter selection and modular expertise without retraining.

Findings

MoLoRA outperforms larger models in reasoning benchmarks.

Specialization with MoLoRA exceeds scale benefits.

Modular adapters enable flexible, efficient inference.

Abstract

Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.