Ensemble Methods for Robust Support Vector Machines using Integer Programming

TL;DR

This paper introduces ensemble methods for robust support vector machines that utilize integer programming to handle data uncertainty, resulting in more stable classification performance under data perturbations.

Contribution

It extends classical robust SVMs with an ensemble approach using integer programming formulations and heuristic algorithms for improved stability against data uncertainties.

Findings

Ensemble methods outperform classical robust SVMs in stability.

Integer programming formulations effectively model adversarial perturbations.

Heuristic algorithms provide efficient solutions with comparable robustness.

Abstract

In this work we study binary classification problems where we assume that our training data is subject to uncertainty, i.e. the precise data points are not known. To tackle this issue in the field of robust machine learning the aim is to develop models which are robust against small perturbations in the training data. We study robust support vector machines (SVM) and extend the classical approach by an ensemble method which iteratively solves a non-robust SVM on different perturbations of the dataset, where the perturbations are derived by an adversarial problem. Afterwards for classification of an unknown data point we perform a majority vote of all calculated SVM solutions. We study three different variants for the adversarial problem, the exact problem, a relaxed variant and an efficient heuristic variant. While the exact and the relaxed variant can be modeled using integer programming formulations, the heuristic one can be implemented by an easy and efficient algorithm. All derived methods are tested on random and realistic datasets and the results indicate that the derived ensemble methods have a much more stable behaviour when changing the protection level compared to the classical robust SVM model.