SDG-MoE: Signed Debate Graph Mixture-of-Experts

TL;DR

SDG-MoE introduces a novel architecture for sparse Mixture-of-Experts that incorporates structured, iterative expert communication to enhance performance and stability, demonstrated by significant perplexity improvements.

Contribution

It proposes a new signed debate graph approach with learned expert interactions and a deliberation process, advancing MoE models with structured expert communication.

Findings

SDG-MoE outperforms baseline models by 19.8% in validation perplexity.

Achieves best external perplexity on WikiText-103, C4, and Paloma datasets.

Theoretical analysis confirms stability and low overhead of the deliberation process.

Abstract

Sparse MoE models achieve a good balance between capacity and compute by routing each token to a small subset of experts. However, in most MoE architectures, once a token is routed, the selected experts process it independently and their outputs are combined via a weighted sum. This leaves open whether enabling communication among them could improve performance. While prior work has raised this question, direct interaction among the active routed experts remains underexplored. In this paper, we propose SDG-MoE (Signed Debate Graph Mixture-of-Experts), a novel architecture that adds a lightweight, iterative deliberation step before final aggregation. SDG-MoE introduces three components: (i) two learned interaction matrices over the active experts, a support graph $A^+$ and a critique graph $A^-$, capturing reinforcing and corrective influences; (ii) a signed message-passing step that updates expert representations before aggregation; and (iii) a disagreement-gated Friedkin-Johnsen-style anchoring that controls deliberation strength while preventing expert drift. Together, these enable a structured deliberation process where interaction strength scales with disagreement and specialization is preserved. We also provide a theoretical analysis establishing stability conditions on expert states and showing that deliberation adds only low-order overhead over the active set. In controlled three-seed pretraining experiments, SDG-MoE improves validation perplexity over both an unsigned graph communication baseline and vanilla MoE, outperforming the strongest baseline by 19.8%, and gives the best external perplexity on WikiText-103, C4, and Paloma among the compared systems.