A Generic Online Parallel Learning Framework for Large Margin Models

TL;DR

This paper introduces a versatile, lock-free online parallel learning framework applicable to large margin models like MIRA and Structured Perceptron, achieving near-linear speedup without accuracy loss.

Contribution

It presents a novel, generic parallel framework for large margin models, extending parallel training beyond stochastic gradient descent methods.

Findings

Near-linear speedup with increased threads

No loss in model accuracy

Applicable to multiple large margin algorithms

Abstract

To speed up the training process, many existing systems use parallel technology for online learning algorithms. However, most research mainly focus on stochastic gradient descent (SGD) instead of other algorithms. We propose a generic online parallel learning framework for large margin models, and also analyze our framework on popular large margin algorithms, including MIRA and Structured Perceptron. Our framework is lock-free and easy to implement on existing systems. Experiments show that systems with our framework can gain near linear speed up by increasing running threads, and with no loss in accuracy.