Spectra: Robust Estimation of Distribution Functions in Networks

TL;DR

Spectra is a distributed algorithm that enables robust, fast, and precise estimation of distribution functions across large networks, effectively handling message loss, node churn, and dynamic data changes.

Contribution

It introduces a novel distributed method for estimating distribution functions that is resilient to network issues and dynamic data, surpassing existing techniques.

Findings

Robustness to message loss and node churn

Fast convergence and high precision in estimates

Effective dynamic adaptation without restarts

Abstract

Distributed aggregation allows the derivation of a given global aggregate property from many individual local values in nodes of an interconnected network system. Simple aggregates such as minima/maxima, counts, sums and averages have been thoroughly studied in the past and are important tools for distributed algorithms and network coordination. Nonetheless, this kind of aggregates may not be comprehensive enough to characterize biased data distributions or when in presence of outliers, making the case for richer estimates of the values on the network. This work presents Spectra, a distributed algorithm for the estimation of distribution functions over large scale networks. The estimate is available at all nodes and the technique depicts important properties, namely: robust when exposed to high levels of message loss, fast convergence speed and fine precision in the estimate. It can also dynamically cope with changes of the sampled local property, not requiring algorithm restarts, and is highly resilient to node churn. The proposed approach is experimentally evaluated and contrasted to a competing state of the art distribution aggregation technique.