Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models

TL;DR

This paper investigates what influences generation quality in decentralized diffusion models, revealing that routing inputs to experts trained on similar data, rather than stability, is key for high-quality outputs.

Contribution

It is the first systematic study showing expert-data alignment, not stability, governs generation quality in decentralized diffusion models.

Findings

Expert-data alignment correlates with higher generation quality.

Sparse routing to data-aligned experts improves output quality.

Full ensemble routing achieves stability but yields poorer quality.

Abstract

Decentralized Diffusion Models (DDMs) route denoising through experts trained independently on disjoint data clusters, which can strongly disagree in their predictions. What governs the quality of generations in such systems? We present the first ever systematic investigation of this question. A priori, the expectation is that minimizing denoising trajectory sensitivity -- minimizing how perturbations amplify during sampling -- should govern generation quality. We demonstrate this hypothesis is incorrect: a stability-quality dissociation. Full ensemble routing, which combines all expert predictions at each step, achieves the most stable sampling dynamics and best numerical convergence while producing the worst generation quality (FID 47.9 vs. 22.6 for sparse Top-2 routing). Instead, we identify expert-data alignment as the governing principle: generation quality depends on routing inputs to experts whose training distribution covers the current denoising state. Across two distinct DDM systems, we validate expert-data alignment using (i) data-cluster distance analysis, confirming sparse routing selects experts with data clusters closest to the current denoising state, and (ii) per-expert analysis, showing selected experts produce more accurate predictions than non-selected ones, and (iii) expert disagreement analysis, showing quality degrades when experts disagree. For DDM deployment, our findings establish that routing should prioritize expert-data alignment over numerical stability metrics.