Slow Learners are Fast

TL;DR

This paper demonstrates that online learning algorithms can be adapted for parallel processing by proving their convergence with delayed updates, addressing the sequential nature of traditional methods.

Contribution

It introduces a theoretical framework showing online learning algorithms remain effective with delayed updates, enabling parallelization on multi-core systems.

Findings

Convergence of online learning with delayed updates is theoretically established.

Parallel online learning can be achieved without sacrificing convergence guarantees.

Abstract

Online learning algorithms have impressive convergence properties when it comes to risk minimization and convex games on very large problems. However, they are inherently sequential in their design which prevents them from taking advantage of modern multi-core architectures. In this paper we prove that online learning with delayed updates converges well, thereby facilitating parallel online learning.