Federated Learning Optimization: A Comparative Study of Data and Model Exchange Strategies in Dynamic Networks

TL;DR

This paper compares data and model exchange strategies in dynamic federated learning to determine the most efficient approach for fast knowledge transfer across devices with varying conditions.

Contribution

It provides a comprehensive analysis of raw data, synthetic data, and model update exchanges, offering insights into optimal strategies for different environments in federated learning.

Findings

Data/model exchange efficiency varies by up to 9.08% across scenarios.

Synthetic data can sometimes outperform raw data in transfer efficiency.

Optimal exchange strategies depend on data distribution and network dynamics.

Abstract

The promise and proliferation of large-scale dynamic federated learning gives rise to a prominent open question - is it prudent to share data or model across nodes, if efficiency of transmission and fast knowledge transfer are the prime objectives. This work investigates exactly that. Specifically, we study the choices of exchanging raw data, synthetic data, or (partial) model updates among devices. The implications of these strategies in the context of foundational models are also examined in detail. Accordingly, we obtain key insights about optimal data and model exchange mechanisms considering various environments with different data distributions and dynamic device and network connections. Across various scenarios that we considered, time-limited knowledge transfer efficiency can differ by up to 9.08\%, thus highlighting the importance of this work.