Convex Loss Functions for Support Vector Machines (SVMs) and Neural Networks

TL;DR

This paper introduces a new convex loss function for SVMs and neural networks, demonstrating improved generalization performance and experimental validation on small datasets for classification and regression tasks.

Contribution

The paper proposes a novel convex loss function that incorporates pattern correlations, with derivations and experimental validation for SVMs and neural networks.

Findings

Up to 2.0% F1 score improvement in classification

1.0% reduction in MSE for regression

Consistent or better generalization measures compared to standard losses

Abstract

We propose a new convex loss for Support Vector Machines, both for the binary classification and for the regression models. Therefore, we show the mathematical derivation of the dual problems and we experiment with them on several small datasets. The minimal dimension of those datasets is due to the difficult scalability of the SVM method to bigger instances. This preliminary study should prove that using pattern correlations inside the loss function could enhance the generalisation performances. Our method consistently achieved comparable or superior performance, with improvements of up to 2.0% in F1 scores for classification tasks and 1.0% reduction in Mean Squared Error (MSE) for regression tasks across various datasets, compared to standard losses. Coherently, results show that generalisation measures are never worse than the standard losses and several times they are better. In our opinion, it should be considered a careful study of this loss, coupled with shallow and deep neural networks. In fact, we present some novel results obtained with those architectures.