CYCLADES: Conflict-free Asynchronous Machine Learning

TL;DR

CYCLADES is a conflict-free asynchronous framework for parallel stochastic optimization that improves speed and efficiency over existing methods by eliminating conflicts and enhancing cache locality.

Contribution

It introduces a novel conflict-free asynchronous parallelization framework that guarantees no conflicts and provides provable speedups for stochastic optimization algorithms.

Findings

Achieves up to 40% speedup over HOGWILD! SGD.

Provides up to 5x speedup over asynchronous variance reduction algorithms.

Outperforms existing asynchronous algorithms on sparse datasets.

Abstract

We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting. CYCLADES is asynchronous during shared model updates, and requires no memory locking mechanisms, similar to HOGWILD!-type algorithms. Unlike HOGWILD!, CYCLADES introduces no conflicts during the parallel execution, and offers a black-box analysis for provable speedups across a large family of algorithms. Due to its inherent conflict-free nature and cache locality, our multi-core implementation of CYCLADES consistently outperforms HOGWILD!-type algorithms on sufficiently sparse datasets, leading to up to 40% speedup gains compared to the HOGWILD! implementation of SGD, and up to 5x gains over asynchronous implementations of variance reduction algorithms.