Learning with Smooth Hinge Losses

TL;DR

This paper introduces two smooth approximations to the Hinge loss, enabling the development of smooth support vector machines that converge faster with modern optimization methods, demonstrated through text classification experiments.

Contribution

It proposes two new smooth Hinge loss functions and a general framework for smoothing convex loss functions, improving optimization efficiency for SVMs.

Findings

Smooth Hinge losses enable faster convergence in SVM training.

Experiments show effectiveness of smooth SVMs in text classification.

Unified framework for smoothing various convex loss functions.

Abstract

Due to the non-smoothness of the Hinge loss in SVM, it is difficult to obtain a faster convergence rate with modern optimization algorithms. In this paper, we introduce two smooth Hinge losses $\psi_G(\alpha;\sigma)$ and $\psi_M(\alpha;\sigma)$ which are infinitely differentiable and converge to the Hinge loss uniformly in $\alpha$ as $\sigma$ tends to $0$. By replacing the Hinge loss with these two smooth Hinge losses, we obtain two smooth support vector machines(SSVMs), respectively. Solving the SSVMs with the Trust Region Newton method (TRON) leads to two quadratically convergent algorithms. Experiments in text classification tasks show that the proposed SSVMs are effective in real-world applications. We also introduce a general smooth convex loss function to unify several commonly-used convex loss functions in machine learning. The general framework provides smooth approximation functions to non-smooth convex loss functions, which can be used to obtain smooth models that can be solved with faster convergent optimization algorithms.