Statistically adaptive learning for a general class of cost functions (SA L-BFGS)

TL;DR

This paper introduces SA L-BFGS, a scalable and flexible optimization method for large-scale machine learning that adapts statistically to efficiently handle trillions of features and billions of examples, enabling rapid experimentation.

Contribution

The paper presents SA L-BFGS, a novel adaptation of batch L-BFGS that uses statistical tools to improve convergence speed and scalability for tera-scale machine learning tasks.

Findings

Outperforms standard practices like Vowpal Wabbit and AllReduce

Achieves near real-time performance on large datasets

Validated with KDD Cup 2012 Tencent data

Abstract

We present a system that enables rapid model experimentation for tera-scale machine learning with trillions of non-zero features, billions of training examples, and millions of parameters. Our contribution to the literature is a new method (SA L-BFGS) for changing batch L-BFGS to perform in near real-time by using statistical tools to balance the contributions of previous weights, old training examples, and new training examples to achieve fast convergence with few iterations. The result is, to our knowledge, the most scalable and flexible linear learning system reported in the literature, beating standard practice with the current best system (Vowpal Wabbit and AllReduce). Using the KDD Cup 2012 data set from Tencent, Inc. we provide experimental results to verify the performance of this method.