Soft-SVM Regression For Binary Classification

TL;DR

This paper introduces Soft-SVM, a new convex relaxation of the hinge loss that bridges logistic regression and SVM, providing an interpretable model that adapts to data separability and improves classification performance.

Contribution

The paper proposes Soft-SVM, a novel exponential family based on a convex relaxation of hinge loss, enhancing interpretability and robustness in binary classification.

Findings

Soft-SVM effectively bridges logistic regression and SVM.

The model adapts to data separability via the softness parameter.

Empirical results show competitive classification and prediction errors.

Abstract

The binomial deviance and the SVM hinge loss functions are two of the most widely used loss functions in machine learning. While there are many similarities between them, they also have their own strengths when dealing with different types of data. In this work, we introduce a new exponential family based on a convex relaxation of the hinge loss function using softness and class-separation parameters. This new family, denoted Soft-SVM, allows us to prescribe a generalized linear model that effectively bridges between logistic regression and SVM classification. This new model is interpretable and avoids data separability issues, attaining good fitting and predictive performance by automatically adjusting for data label separability via the softness parameter. These results are confirmed empirically through simulations and case studies as we compare regularized logistic, SVM, and Soft-SVM regressions and conclude that the proposed model performs well in terms of both classification and prediction errors.