Linearized Additive Classifiers

TL;DR

This paper introduces efficient additive classifiers using spline basis and orthogonal embeddings, achieving comparable accuracy to kernel SVMs with significantly reduced training time and memory usage.

Contribution

It adapts penalized spline formulations for additive classifiers and proposes new orthogonal basis embeddings for efficient learning.

Findings

Additive classifiers match kernel SVM accuracy on large datasets.

Proposed methods are 2 orders of magnitude faster to train.

Spline basis closely approximates histogram intersection kernel.

Abstract

We revisit the additive model learning literature and adapt a penalized spline formulation due to Eilers and Marx, to train additive classifiers efficiently. We also propose two new embeddings based two classes of orthogonal basis with orthogonal derivatives, which can also be used to efficiently learn additive classifiers. This paper follows the popular theme in the current literature where kernel SVMs are learned much more efficiently using a approximate embedding and linear machine. In this paper we show that spline basis are especially well suited for learning additive models because of their sparsity structure and the ease of computing the embedding which enables one to train these models in an online manner, without incurring the memory overhead of precomputing the storing the embeddings. We show interesting connections between B-Spline basis and histogram intersection kernel and show that for a particular choice of regularization and degree of the B-Splines, our proposed learning algorithm closely approximates the histogram intersection kernel SVM. This enables one to learn additive models with almost no memory overhead compared to fast a linear solver, such as LIBLINEAR, while being only 5-6X slower on average. On two large scale image classification datasets, MNIST and Daimler Chrysler pedestrians, the proposed additive classifiers are as accurate as the kernel SVM, while being two orders of magnitude faster to train.