Mosaic Learning: A Framework for Decentralized Learning with Model Fragmentation

TL;DR

Mosaic Learning introduces a decentralized framework that fragments models to improve communication efficiency and accuracy, demonstrating state-of-the-art convergence and significant accuracy gains across multiple tasks.

Contribution

It proposes a novel model fragmentation approach for decentralized learning, enhancing convergence and accuracy without increasing communication costs.

Findings

Up to 12 percentage points higher node-level test accuracy

State-of-the-art worst-case convergence rate

Effective leverage of parameter correlation in models

Abstract

Decentralized learning (DL) enables collaborative machine learning (ML) without a central server, making it suitable for settings where training data cannot be centrally hosted. We introduce Mosaic Learning, a DL framework that decomposes models into fragments and disseminates them independently across the network. Fragmentation reduces redundant communication across correlated parameters and enables more diverse information propagation without increasing communication cost. We theoretically show that Mosaic Learning (i) shows state-of-the-art worst-case convergence rate, and (ii) leverages parameter correlation in an ML model, improving contraction by reducing the highest eigenvalue of a simplified system. We empirically evaluate Mosaic Learning on four learning tasks and observe up to 12 percentage points higher node-level test accuracy compared to epidemic learning (EL), a state-of-the-art baseline. In summary, Mosaic Learning improves DL performance without sacrificing its utility or efficiency, and positions itself as a new DL standard.