Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

TL;DR

This paper introduces DDOME, a method for dynamically specializing experts in decentralized settings like federated learning, improving accuracy and personalization by joint training of gating functions and experts.

Contribution

The paper proposes DDOME, a novel framework for joint gating-expert training in decentralized environments, enabling dynamic expert specialization and personalization.

Findings

DDOME improves accuracy by 4-24% over state-of-the-art FL baselines.

It achieves personalized expert subset selection on-the-fly.

Theoretical analysis confirms joint training is key for expert specialization.

Abstract

Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference costs and data heterogeneity, we study how joint training of gating functions and experts can dynamically allocate domain-specific expertise across multiple underlying data distributions. As an outcome of our framework, we develop an instance tailored specifically to decentralized training scenarios, introducing \textit{Dynamically Decentralized Orchestration of MoEs} or \texttt{DDOME}. \texttt{DDOME} leverages heterogeneity emerging from distributional shifts across decentralized data sources to specialize experts dynamically. By integrating a pretrained common expert to inform a gating function, \texttt{DDOME} achieves personalized expert subset selection on-the-fly, facilitating just-in-time personalization. We empirically validate \texttt{DDOME} within a Federated Learning (FL) context: \texttt{DDOME} attains from 4\% up to an 24\% accuracy improvement over state-of-the-art FL baselines in image and text classification tasks, while maintaining competitive zero-shot generalization capabilities. Furthermore, we provide theoretical insights confirming that the joint gating-experts training is critical for achieving meaningful expert specialization.