Optimizing the Optimal Weighted Average: Efficient Distributed Sparse Classification

TL;DR

This paper introduces ACOWA, a new distributed algorithm for sparse logistic regression that improves approximation quality and accuracy with minimal additional communication, addressing challenges in high-dimensional data settings.

Contribution

ACOWA adds an extra communication round to the optimal weighted average method, significantly enhancing approximation quality in distributed sparse classification.

Findings

ACOWA achieves higher accuracy than existing distributed algorithms.

ACOWA produces solutions closer to the empirical risk minimizer.

The method maintains efficiency with only a minor runtime increase.

Abstract

While distributed training is often viewed as a solution to optimizing linear models on increasingly large datasets, inter-machine communication costs of popular distributed approaches can dominate as data dimensionality increases. Recent work on non-interactive algorithms shows that approximate solutions for linear models can be obtained efficiently with only a single round of communication among machines. However, this approximation often degenerates as the number of machines increases. In this paper, building on the recent optimal weighted average method, we introduce a new technique, ACOWA, that allows an extra round of communication to achieve noticeably better approximation quality with minor runtime increases. Results show that for sparse distributed logistic regression, ACOWA obtains solutions that are more faithful to the empirical risk minimizer and attain substantially higher accuracy than other distributed algorithms.