Distributed Estimation, Information Loss and Exponential Families

TL;DR

This paper analyzes a communication-efficient distributed learning framework for probabilistic models, revealing how the deviation from exponential family distributions affects estimation accuracy and comparing combination methods.

Contribution

It establishes the theoretical lower bound of estimation error based on distribution family properties and compares KL-divergence-based and linear combination methods.

Findings

KL-divergence-based combination achieves lower error bounds.

Full exponential family structure minimizes information loss.

KL method outperforms linear combination in practical scenarios.

Abstract

Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs to achieve the best possible approximation to the global MLE given the whole dataset. We study this framework's statistical properties, showing that the efficiency loss compared to the global setting relates to how much the underlying distribution families deviate from full exponential families, drawing connection to the theory of information loss by Fisher, Rao and Efron. We show that the "full-exponential-family-ness" represents the lower bound of the error rate of arbitrary combinations of local MLEs, and is achieved by a KL-divergence-based combination method but not by a more common linear combination method. We also study the empirical properties of both methods, showing that the KL method significantly outperforms linear combination in practical settings with issues such as model misspecification, non-convexity, and heterogeneous data partitions.