Hybrid-DCA: A Double Asynchronous Approach for Stochastic Dual Coordinate Ascent

TL;DR

This paper introduces Hybrid-DCA, a double asynchronous stochastic dual coordinate ascent algorithm designed for multi-core clusters, which achieves faster convergence and better scalability on large datasets than existing methods.

Contribution

It proposes a novel hybrid asynchronous framework for SDCA on multi-core clusters, combining local asynchronous updates with across-node aggregation, and proves its convergence with empirical validation.

Findings

Achieves linear convergence rate for smooth convex loss functions.

Scales better and runs faster than existing shared-memory and distributed-memory algorithms.

Successfully handles large datasets, such as 280 GB, significantly faster than prior methods.

Abstract

In prior works, stochastic dual coordinate ascent (SDCA) has been parallelized in a multi-core environment where the cores communicate through shared memory, or in a multi-processor distributed memory environment where the processors communicate through message passing. In this paper, we propose a hybrid SDCA framework for multi-core clusters, the most common high performance computing environment that consists of multiple nodes each having multiple cores and its own shared memory. We distribute data across nodes where each node solves a local problem in an asynchronous parallel fashion on its cores, and then the local updates are aggregated via an asynchronous across-node update scheme. The proposed double asynchronous method converges to a global solution for $L$-Lipschitz continuous loss functions, and at a linear convergence rate if a smooth convex loss function is used. Extensive empirical comparison has shown that our algorithm scales better than the best known shared-memory methods and runs faster than previous distributed-memory methods. Big datasets, such as one of 280 GB from the LIBSVM repository, cannot be accommodated on a single node and hence cannot be solved by a parallel algorithm. For such a dataset, our hybrid algorithm takes 30 seconds to achieve a duality gap of $10^{-6}$ on 16 nodes each using 8 cores, which is significantly faster than the best known distributed algorithms, such as CoCoA+, that take more than 300 seconds on 16 nodes.