Sublinear Optimization for Machine Learning

TL;DR

This paper introduces sublinear-time algorithms for key machine learning optimization problems, extending to kernelized versions, with nearly optimal running times and efficient semi-streaming implementations.

Contribution

It presents novel sampling and multiplicative update algorithms for sublinear approximation of ML problems, including kernelized variants, with proven near-optimal bounds.

Findings

Algorithms achieve sublinear time for training classifiers and finding minimum enclosing balls.

Extensions to kernelized problems like SVDD and SVM are demonstrated.

First low pass polylogarithmic space algorithms in semi-streaming setting.

Abstract

We give sublinear-time approximation algorithms for some optimization problems arising in machine learning, such as training linear classifiers and finding minimum enclosing balls. Our algorithms can be extended to some kernelized versions of these problems, such as SVDD, hard margin SVM, and L2-SVM, for which sublinear-time algorithms were not known before. These new algorithms use a combination of a novel sampling techniques and a new multiplicative update algorithm. We give lower bounds which show the running times of many of our algorithms to be nearly best possible in the unit-cost RAM model. We also give implementations of our algorithms in the semi-streaming setting, obtaining the first low pass polylogarithmic space and sublinear time algorithms achieving arbitrary approximation factor.