Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments

TL;DR

Mosaic introduces a data-free knowledge distillation framework that leverages local generative models and a Mixture-of-Experts architecture to improve federated learning performance amid heterogeneity, without sharing real data.

Contribution

The paper presents a novel data-free distillation method using local generative models and a Mixture-of-Experts approach tailored for heterogeneous federated environments.

Findings

Outperforms state-of-the-art methods on image classification benchmarks.

Effectively handles both model and data heterogeneity.

Enhances global model performance without sharing real data.

Abstract

Federated Learning (FL) is a decentralized machine learning paradigm that enables clients to collaboratively train models while preserving data privacy. However, the coexistence of model and data heterogeneity gives rise to inconsistent representations and divergent optimization dynamics across clients, ultimately hindering robust global performance. To transcend these challenges, we propose Mosaic, a novel data-free knowledge distillation framework tailored for heterogeneous distributed environments. Mosaic first trains local generative models to approximate each client's personalized distribution, enabling synthetic data generation that safeguards privacy through strict separation from real data. Subsequently, Mosaic forms a Mixture-of-Experts (MoE) from client models based on their specialized knowledge, and distills it into a global model using the generated data. To further enhance the MoE architecture, Mosaic integrates expert predictions via a lightweight meta model trained on a few representative prototypes. Extensive experiments on standard image classification benchmarks demonstrate that Mosaic consistently outperforms state-of-the-art approaches under both model and data heterogeneity. The source code has been published at https://github.com/Wings-Of-Disaster/Mosaic.