CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models

TL;DR

CoDream introduces a novel federated learning framework that exchanges knowledge in the data space rather than model parameters, enabling model-agnostic, scalable, and privacy-preserving collaborative learning.

Contribution

It proposes a new approach to federated learning by optimizing and sharing data-derived knowledge instead of model parameters, supporting heterogeneous models and enhancing privacy.

Findings

Achieves competitive performance on standard FL tasks

Supports model-agnostic and scalable federated learning

Preserves privacy through secure data space sharing

Abstract

Federated Learning (FL) enables collaborative optimization of machine learning models across decentralized data by aggregating model parameters. Our approach extends this concept by aggregating "knowledge" derived from models, instead of model parameters. We present a novel framework called CoDream, where clients collaboratively optimize randomly initialized data using federated optimization in the input data space, similar to how randomly initialized model parameters are optimized in FL. Our key insight is that jointly optimizing this data can effectively capture the properties of the global data distribution. Sharing knowledge in data space offers numerous benefits: (1) model-agnostic collaborative learning, i.e., different clients can have different model architectures; (2) communication that is independent of the model size, eliminating scalability concerns with model parameters; (3) compatibility with secure aggregation, thus preserving the privacy benefits of federated learning; (4) allowing of adaptive optimization of knowledge shared for personalized learning. We empirically validate CoDream on standard FL tasks, demonstrating competitive performance despite not sharing model parameters. Our code: https://mitmedialab.github.io/codream.github.io/