Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers

TL;DR

This paper introduces routing signatures in sparse MoE transformers, revealing that expert routing patterns are task-conditioned and can be used for task classification, highlighting a structured aspect of conditional computation.

Contribution

The authors propose routing signatures to analyze expert activation patterns, demonstrating their task-conditioned nature and their effectiveness in classifying tasks in sparse MoE models.

Findings

Routing signatures are highly similar for prompts from the same task.

A classifier trained on routing signatures achieves over 92% accuracy in task classification.

Task structure becomes more evident in deeper layers of the model.

Abstract

Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models through conditional computation, yet the routing mechanisms responsible for expert selection remain poorly understood. In this work, we introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt, and use them to study whether MoE routing exhibits task-conditioned structure. Using OLMoE-1B-7B-0125-Instruct as an empirical testbed, we show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity. Within-category routing similarity (0.8435 +/- 0.0879) significantly exceeds across-category similarity (0.6225 +/- 0.1687), corresponding to Cohen's d = 1.44. A logistic regression classifier trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification. To ensure statistical validity, we introduce permutation and load-balancing baselines and show that the observed separation is not explained by sparsity or balancing constraints alone. We further analyze layer-wise signal strength and low-dimensional projections of routing signatures, finding that task structure becomes increasingly apparent in deeper layers. These results suggest that routing in sparse transformers is not merely a balancing mechanism, but a measurable task-sensitive component of conditional computation. We release MOE-XRAY, a lightweight toolkit for routing telemetry and analysis.