DistDD: Distributed Data Distillation Aggregation through Gradient Matching

TL;DR

DistDD introduces a one-time data distillation method for federated learning that reduces communication costs, preserves privacy, and enables efficient model tuning and neural architecture search, especially in complex data scenarios.

Contribution

The paper presents a novel data distillation approach for federated learning that minimizes communication and supports multiple downstream tasks without iterative model updates.

Findings

Effective in non-i.i.d. and mislabeled data scenarios

Reduces communication costs significantly

Enables neural architecture search without full FL retraining

Abstract

In this paper, we introduce DistDD, a novel approach within the federated learning framework that reduces the need for repetitive communication by distilling data directly on clients' devices. Unlike traditional federated learning that requires iterative model updates across nodes, DistDD facilitates a one-time distillation process that extracts a global distilled dataset, maintaining the privacy standards of federated learning while significantly cutting down communication costs. By leveraging the DistDD's distilled dataset, the developers of the FL can achieve just-in-time parameter tuning and neural architecture search over FL without repeating the whole FL process multiple times. We provide a detailed convergence proof of the DistDD algorithm, reinforcing its mathematical stability and reliability for practical applications. Our experiments demonstrate the effectiveness and robustness of DistDD, particularly in non-i.i.d. and mislabeled data scenarios, showcasing its potential to handle complex real-world data challenges distinctively from conventional federated learning methods. We also evaluate DistDD's application in the use case and prove its effectiveness and communication-savings in the NAS use case.