F-SVM: Combination of Feature Transformation and SVM Learning via Convex Relaxation

TL;DR

This paper introduces F-SVM, a convex model that jointly learns feature transformation and SVM classifier by approximating the radius of the minimum enclosing ball, leading to improved classification performance.

Contribution

It proposes a novel convex radius-margin SVM model with an efficient alternating minimization approach for joint feature transformation and classification.

Findings

F-SVM outperforms standard SVM on multiple datasets.

The model effectively integrates radius and margin for better generalization.

Extension with kernel PCA enables nonlinear transformation learning.

Abstract

The generalization error bound of support vector machine (SVM) depends on the ratio of radius and margin, while standard SVM only considers the maximization of the margin but ignores the minimization of the radius. Several approaches have been proposed to integrate radius and margin for joint learning of feature transformation and SVM classifier. However, most of them either require the form of the transformation matrix to be diagonal, or are non-convex and computationally expensive. In this paper, we suggest a novel approximation for the radius of minimum enclosing ball (MEB) in feature space, and then propose a convex radius-margin based SVM model for joint learning of feature transformation and SVM classifier, i.e., F-SVM. An alternating minimization method is adopted to solve the F-SVM model, where the feature transformation is updatedvia gradient descent and the classifier is updated by employing the existing SVM solver. By incorporating with kernel principal component analysis, F-SVM is further extended for joint learning of nonlinear transformation and classifier. Experimental results on the UCI machine learning datasets and the LFW face datasets show that F-SVM outperforms the standard SVM and the existing radius-margin based SVMs, e.g., RMM, R-SVM+ and R-SVM+{\mu}.