Parallel Online Learning

TL;DR

This paper investigates parallel online learning, analyzing the impact of delayed updates caused by parallelization, and explores feature sharding architectures to balance delay, parallelism, and learning performance.

Contribution

It introduces a feature sharding approach for parallel online learning and analyzes tradeoffs between delay, parallelism, and empirical effectiveness.

Findings

Delayed updates can significantly impair learning performance.

Feature sharding offers a tradeoff between delay and empirical accuracy.

Preliminary empirical results demonstrate potential benefits of the proposed architectures.

Abstract

In this work we study parallelization of online learning, a core primitive in machine learning. In a parallel environment all known approaches for parallel online learning lead to delayed updates, where the model is updated using out-of-date information. In the worst case, or when examples are temporally correlated, delay can have a very adverse effect on the learning algorithm. Here, we analyze and present preliminary empirical results on a set of learning architectures based on a feature sharding approach that present various tradeoffs between delay, degree of parallelism, representation power and empirical performance.