Proximal Operator and Optimality Conditions for Ramp Loss SVM

TL;DR

This paper develops a new optimality framework for ramp loss SVMs by introducing the proximal operator and establishing P-stationarity, providing a comprehensive understanding of their local minima and support vectors.

Contribution

It introduces the proximal operator for ramp loss and proves P-stationarity as the first-order necessary and sufficient condition for local minima in ramp loss SVMs.

Findings

P-stationarity characterizes local minima of $L_r$-SVM.

All $L_r$ support vectors lie on support hyperplanes.

Proximal operator analysis deepens the understanding of ramp loss SVMs.

Abstract

Support vector machines with ramp loss (dubbed as $L_r$-SVM) have attracted wide attention due to the boundedness of ramp loss. However, the corresponding optimization problem is non-convex and the given Karush-Kuhn-Tucker (KKT) conditions are only the necessary conditions. To enrich the optimality theory of $L_r$-SVM and go deep into its statistical nature, we first introduce and analyze the proximal operator for ramp loss, and then establish a stronger optimality conditions: P-stationarity, which is proved to be the first-order necessary and sufficient conditions for local minimizer of $L_r$-SVM. Finally, we define the $L_r$ support vectors based on the concept of P-stationary point, and show that all $L_r$ support vectors fall into the support hyperplanes, which possesses the same feature as the one of hard margin SVM.