Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization

TL;DR

Anytime Minibatch is a distributed optimization method that mitigates stragglers by allowing variable local computation times and fixed communication periods, leading to faster convergence in heterogeneous environments.

Contribution

The paper introduces Anytime Minibatch, a novel online distributed optimization technique that effectively exploits stragglers to improve speed without sacrificing accuracy.

Findings

Up to 1.5x faster in Amazon EC2 environments.

Up to 5x faster with high variability in node performance.

Convergence guarantees and wall time performance analysis.

Abstract

Distributed optimization is vital in solving large-scale machine learning problems. A widely-shared feature of distributed optimization techniques is the requirement that all nodes complete their assigned tasks in each computational epoch before the system can proceed to the next epoch. In such settings, slow nodes, called stragglers, can greatly slow progress. To mitigate the impact of stragglers, we propose an online distributed optimization method called Anytime Minibatch. In this approach, all nodes are given a fixed time to compute the gradients of as many data samples as possible. The result is a variable per-node minibatch size. Workers then get a fixed communication time to average their minibatch gradients via several rounds of consensus, which are then used to update primal variables via dual averaging. Anytime Minibatch prevents stragglers from holding up the system without wasting the work that stragglers can complete. We present a convergence analysis and analyze the wall time performance. Our numerical results show that our approach is up to 1.5 times faster in Amazon EC2 and it is up to five times faster when there is greater variability in compute node performance.