Janus: Disaggregating Attention and Experts for Scalable MoE Inference

TL;DR

JANUS is a scalable, resource-efficient MoE inference system that disaggregates attention and expert layers, enabling independent resource management and improved performance.

Contribution

It introduces a novel disaggregation approach, an adaptive communication mechanism, and a microsecond-scale activation scheduler for efficient MoE inference.

Findings

Up to 4.7x throughput improvement over baselines

Reduces inference latency by balancing activated experts

Minimizes GPU cost while meeting latency SLOs

Abstract

Serving large Mixture-of-Experts (MoE) models is challenging because of their large memory footprints, heterogeneous resource demands, and highly dynamic inference workloads. Most existing MoE inference systems deploy the entire model as a monolithic unit, forcing attention and MoE layers to share the same resource configuration despite their different scaling behaviors and resource bottlenecks. Such coarse-grained provisioning leads to resource inefficiency and suboptimal performance. We present JANUS, a scalable and resource-efficient MoE inference system built around three key principles. First, JANUS disaggregates attention and MoE layers onto separate GPU worker pools, enabling independent resource provisioning for the two layer types, and uses an adaptive two-phase communication mechanism for low-latency data exchange. Second, because MoE-layer execution is often memory-bound and highly sensitive to activated-expert imbalance, JANUS introduces a lightweight, microsecond-scale activation scheduler that balances per-layer activated experts across MoE instances to reduce inference latency. Third, JANUS employs a fine-grained, SLO-aware resource scaling scheme that jointly selects attention resources, MoE resources, and expert placement to minimize GPU cost under token-level SLOs. Evaluation shows that JANUS improves per-GPU throughput by up to 4.7x over state-of-the-art MoE inference baselines while satisfying token-level latency SLOs.