Avoiding Communication in Logistic Regression

TL;DR

This paper introduces a communication-avoiding variant of SGD for logistic regression, reducing communication frequency and achieving significant speedups on high-performance clusters without sacrificing accuracy.

Contribution

It proposes a novel communication-avoiding SGD technique that reorganizes computations to reduce communication, with theoretical bounds and practical speedup results.

Findings

Achieves up to 4.97x speedup on high-performance clusters.

Maintains convergence behavior and accuracy of standard SGD.

Provides theoretical bounds on flops, bandwidth, and latency.

Abstract

Stochastic gradient descent (SGD) is one of the most widely used optimization methods for solving various machine learning problems. SGD solves an optimization problem by iteratively sampling a few data points from the input data, computing gradients for the selected data points, and updating the solution. However, in a parallel setting, SGD requires interprocess communication at every iteration. We introduce a new communication-avoiding technique for solving the logistic regression problem using SGD. This technique re-organizes the SGD computations into a form that communicates every $s$ iterations instead of every iteration, where $s$ is a tuning parameter. We prove theoretical flops, bandwidth, and latency upper bounds for SGD and its new communication-avoiding variant. Furthermore, we show experimental results that illustrate that the new Communication-Avoiding SGD (CA-SGD) method can achieve speedups of up to $4.97\times$ on a high-performance Infiniband cluster without altering the convergence behavior or accuracy.