The Stochastic Gradient Descent for the Primal L1-SVM Optimization Revisited

TL;DR

This paper revisits stochastic gradient methods for primal L1-SVM optimization, revealing connections to perceptron algorithms, establishing bounds for dual variables, and proposing mechanisms that improve convergence and performance.

Contribution

It introduces a novel perspective linking stochastic gradient descent for L1-SVMs to perceptron algorithms and develops mechanisms for better convergence and stopping criteria.

Findings

Dual variables obey box constraints after each cycle

The dual Lagrangian provides a lower bound on the primal objective

Experimental results show improved algorithm performance

Abstract

We reconsider the stochastic (sub)gradient approach to the unconstrained primal L1-SVM optimization. We observe that if the learning rate is inversely proportional to the number of steps, i.e., the number of times any training pattern is presented to the algorithm, the update rule may be transformed into the one of the classical perceptron with margin in which the margin threshold increases linearly with the number of steps. Moreover, if we cycle repeatedly through the possibly randomly permuted training set the dual variables defined naturally via the expansion of the weight vector as a linear combination of the patterns on which margin errors were made are shown to obey at the end of each complete cycle automatically the box constraints arising in dual optimization. This renders the dual Lagrangian a running lower bound on the primal objective tending to it at the optimum and makes available an upper bound on the relative accuracy achieved which provides a meaningful stopping criterion. In addition, we propose a mechanism of presenting the same pattern repeatedly to the algorithm which maintains the above properties. Finally, we give experimental evidence that algorithms constructed along these lines exhibit a considerably improved performance.