A Novel Loss Function-based Support Vector Machine for Binary Classification

TL;DR

This paper introduces a new loss function-based SVM that improves classification by better penalizing correctly classified samples, enhancing generalization and robustness.

Contribution

It proposes a novel Slide loss function for SVM, derives optimality conditions, and develops an efficient ADMM algorithm with convergence analysis.

Findings

Demonstrates improved robustness on real datasets

Shows enhanced generalization ability

Provides a fast optimization method for the new SVM

Abstract

The previous support vector machine(SVM) including $0/1$ loss SVM, hinge loss SVM, ramp loss SVM, truncated pinball loss SVM, and others, overlooked the degree of penalty for the correctly classified samples within the margin. This oversight affects the generalization ability of the SVM classifier to some extent. To address this limitation, from the perspective of confidence margin, we propose a novel Slide loss function ($\ell_s$) to construct the support vector machine classifier($\ell_s$-SVM). By introducing the concept of proximal stationary point, and utilizing the property of Lipschitz continuity, we derive the first-order optimality conditions for $\ell_s$-SVM. Based on this, we define the $\ell_s$ support vectors and working set of $\ell_s$-SVM. To efficiently handle $\ell_s$-SVM, we devise a fast alternating direction method of multipliers with the working set ($\ell_s$-ADMM), and provide the convergence analysis. The numerical experiments on real world datasets confirm the robustness and effectiveness of the proposed method.