ReBoot: Distributed statistical learning via refitting bootstrap samples

TL;DR

ReBoot is a one-shot distributed learning algorithm that refits models on bootstrap samples, achieving full-sample statistical rates with minimal communication, and is extended to federated CNN training with superior early-round performance.

Contribution

This paper introduces ReBoot, a novel distributed learning method that refits bootstrap samples with only one communication round, and extends it to federated CNNs with FedReBoot.

Findings

ReBoot achieves the full-sample statistical rate for GLM and phase retrieval.

ReBoot's bias rate is sharper than model averaging, indicating higher data split tolerance.

FedReBoot outperforms FedAvg in early communication rounds for image classification.

Abstract

In this paper, we propose a one-shot distributed learning algorithm via refitting bootstrap samples, which we refer to as ReBoot. ReBoot refits a new model to mini-batches of bootstrap samples that are continuously drawn from each of the locally fitted models. It requires only one round of communication of model parameters without much memory. Theoretically, we analyze the statistical error rate of ReBoot for generalized linear models (GLM) and noisy phase retrieval, which represent convex and non-convex problems, respectively. In both cases, ReBoot provably achieves the full-sample statistical rate. In particular, we show that the systematic bias of ReBoot, the error that is independent of the number of subsamples (i.e., the number of sites), is $O(n ^ {-2})$ in GLM, where $n$ is the subsample size (the sample size of each local site). This rate is sharper than that of model parameter averaging and its variants, implying the higher tolerance of ReBoot with respect to data splits to maintain the full-sample rate. Our simulation study demonstrates the statistical advantage of ReBoot over competing methods. Finally, we propose FedReBoot, an iterative version of ReBoot, to aggregate convolutional neural networks for image classification. FedReBoot exhibits substantial superiority over Federated Averaging (FedAvg) within early rounds of communication.