On Robust Aggregation for Distributed Data

TL;DR

This paper proposes a robust Huber-type aggregation method for distributed data analysis, effectively handling contamination in local estimates and maintaining statistical efficiency comparable to pooled data analysis.

Contribution

It introduces a novel Huber-type aggregation technique for M-estimators in distributed systems, with theoretical guarantees and contamination detection capabilities.

Findings

Achieves the same convergence rate as pooled data analysis.

Provides asymptotic normality for inference.

Validated through extensive simulations and airline data application.

Abstract

When data are stored across multiple locations, directly pooling all the data together for statistical analysis may be impossible due to communication costs and privacy concerns. Distributed computing systems allow the analysis of such data, by getting local servers to separately process their own statistical analyses and using a central processor to aggregate the local statistical results. Naive aggregation of local statistics using simple or weighted averages, is vulnerable to contamination within a distributed computing system. This paper develops and investigates a Huber-type aggregation method for locally computed M-estimators to handle contamination in the local estimates. Our implementation of this aggregation method requires estimating the asymptotic variance-covariance matrix of the M-estimator, which we accomplish using a robust spatial median approach. Theoretically, the Huber-type aggregation achieves the same convergence rate as if all the data were pooled. We establish its asymptotic normality for making inferences, including justifying a two-step approach for detecting contamination in the distributed computing system. Extensive simulation studies are conducted to validate the theoretical results and the usefulness of our proposed approach is demonstrated on U.S. airline data.