An Information-theoretic Method for Collaborative Distributed Learning with Limited Communication

TL;DR

This paper introduces an information-theoretic approach to optimize limited communication in distributed learning, focusing on selecting and allocating eigenvector-based statistics to minimize expected population risk.

Contribution

It proposes a novel eigenvector-based method for efficient statistic transmission in distributed learning with limited communication, including analytical solutions and an algorithm for general cases.

Findings

Eigenvector-based statistics improve learning performance.

Analytical solutions for single and two-node cases.

An efficient algorithm for general node partitioning.

Abstract

In this paper, we study the information transmission problem under the distributed learning framework, where each worker node is merely permitted to transmit a $m$-dimensional statistic to improve learning results of the target node. Specifically, we evaluate the corresponding expected population risk (EPR) under the regime of large sample sizes. We prove that the performance can be enhanced since the transmitted statistics contribute to estimating the underlying distribution under the mean square error measured by the EPR norm matrix. Accordingly, the transmitted statistics correspond to the eigenvectors of this matrix, and the desired transmission allocates these eigenvectors among the statistics such that the EPR is minimal. Moreover, we provide the analytical solution of the desired statistics for single-node and two-node transmission, where a geometrical interpretation is given to explain the eigenvector selection. For the general case, an efficient algorithm that can output the allocation solution is developed based on the node partitions.