Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts

TL;DR

This paper investigates how vision Mixture-of-Experts models encode information, revealing that expert specialization extends beyond simple category routing to broader visual and semantic features, with stable and interpretable patterns.

Contribution

The study introduces expert-level analyses using neuroscience-inspired tools, demonstrating that expert tuning involves continuous features and is stable across initializations, beyond traditional routing interpretations.

Findings

Expert partitioning is dominated by animate-inanimate distinction.

Experts show broader tuning to continuous visual and semantic features.

Expert specialization is stable across different initializations.

Abstract

Mixture-of-Experts (MoE) models are often interpreted by analysing which categories are routed to which experts. However, routing alone does not reveal what each expert actually encodes. We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and characterise expert specialisation using tools from visual neuroscience. Extending from gating-level to expert-level analyses, we measure per-expert category separability, and per-expert tuning using the most exciting inputs. Extending from category-level to feature-level explanations, we interpret tuning via semantic dimensions derived from a dataset of human behavioural judgements (THINGS). Finally, we use tuning and representational similarity analysis to assess the stability of expertise-allocation across independent initialisations. We find that an animate-inanimate distinction dominates expert partitioning, apparent from gating through to expert readout, and is stable across independently trained models. Although routing statistics suggest relatively sparse, categorical preferences, expert analyses reveal broader tuning to continuous visual and semantic dimensions that extend beyond category boundaries. Experts exhibit similar category-separability to one another, despite distinct feature tuning, demonstrating the explanatory benefits of moving beyond category-level analyses. Together, these results show that expert specialisation in vision MoEs extends well beyond category routing and is better understood by probing fine-grained expert-level tuning and representational structure.