Efficient Distributed Learning with Sparsity

TL;DR

This paper introduces an efficient distributed sparse learning method that reduces communication costs and matches centralized estimation accuracy, suitable for high-dimensional regression and classification tasks.

Contribution

It presents a novel distributed approach that requires minimal communication and achieves estimation error bounds comparable to centralized methods.

Findings

Matches centralized estimation error bounds within constant communication rounds

Demonstrates strong performance on high-dimensional regression tasks

Effective on both simulated and real-world datasets

Abstract

We propose a novel, efficient approach for distributed sparse learning in high-dimensions, where observations are randomly partitioned across machines. Computationally, at each round our method only requires the master machine to solve a shifted ell_1 regularized M-estimation problem, and other workers to compute the gradient. In respect of communication, the proposed approach provably matches the estimation error bound of centralized methods within constant rounds of communications (ignoring logarithmic factors). We conduct extensive experiments on both simulated and real world datasets, and demonstrate encouraging performances on high-dimensional regression and classification tasks.