Least Squares Revisited: Scalable Approaches for Multi-class Prediction

TL;DR

This paper introduces simple, scalable, and parameter-free iterative least-squares algorithms for multi-class and multi-label prediction that outperform first-order methods both theoretically and empirically.

Contribution

It presents new iterative least-squares algorithms with convergence guarantees and demonstrates their practical efficiency and accuracy on large-scale datasets.

Findings

Algorithms outperform first-order methods in practice

Achieve significant speedups over Liblinear and Vowpal Wabbit

Attain state-of-the-art accuracy on MNIST and CIFAR-10

Abstract

This work provides simple algorithms for multi-class (and multi-label) prediction in settings where both the number of examples n and the data dimension d are relatively large. These robust and parameter free algorithms are essentially iterative least-squares updates and very versatile both in theory and in practice. On the theoretical front, we present several variants with convergence guarantees. Owing to their effective use of second-order structure, these algorithms are substantially better than first-order methods in many practical scenarios. On the empirical side, we present a scalable stagewise variant of our approach, which achieves dramatic computational speedups over popular optimization packages such as Liblinear and Vowpal Wabbit on standard datasets (MNIST and CIFAR-10), while attaining state-of-the-art accuracies.