Implicitly Maximizing Margins with the Hinge Loss

TL;DR

This paper introduces a modified hinge loss for neural networks that guarantees faster convergence to the maximum margin in classification tasks, outperforming traditional exponential loss functions.

Contribution

It extends the hinge loss by assigning gradients at critical points, achieving faster convergence rates for linear classifiers and demonstrating similar benefits in ReLU networks.

Findings

Convergence rate to max-margin is (1/t) for the new loss.

Empirical results show improved margin convergence in ReLU networks.

The method outperforms logistic loss in convergence speed.

Abstract

A new loss function is proposed for neural networks on classification tasks which extends the hinge loss by assigning gradients to its critical points. We will show that for a linear classifier on linearly separable data with fixed step size, the margin of this modified hinge loss converges to the $\ell_2$ max-margin at the rate of $\mathcal{O}( 1/t )$. This rate is fast when compared with the $\mathcal{O}(1/\log t)$ rate of exponential losses such as the logistic loss. Furthermore, empirical results suggest that this increased convergence speed carries over to ReLU networks.