Wasserstein Distributionally Robust Multiclass Support Vector Machine

TL;DR

This paper introduces a Wasserstein distributionally robust multiclass SVM that effectively handles uncertain and imbalanced data, providing a convex reformulation and demonstrating superior performance over existing models.

Contribution

The paper develops a novel Wasserstein distributionally robust multiclass SVM with a tractable convex reformulation and a kernel extension, addressing label uncertainty and class imbalance.

Findings

Outperforms state-of-the-art OVA models on imbalanced datasets

Provides a convex reformulation of the worst-case risk problem

Demonstrates effectiveness of the kernel version and scalability with a subgradient method

Abstract

We study the problem of multiclass classification for settings where data features $\mathbf{x}$ and their labels $\mathbf{y}$ are uncertain. We identify that distributionally robust one-vs-all (OVA) classifiers often struggle in settings with imbalanced data. To address this issue, we use Wasserstein distributionally robust optimization to develop a robust version of the multiclass support vector machine (SVM) characterized by the Crammer-Singer (CS) loss. First, we prove that the CS loss is bounded from above by a Lipschitz continuous function for all $\mathbf{x} \in \mathcal{X}$ and $\mathbf{y} \in \mathcal{Y}$, then we exploit strong duality results to express the dual of the worst-case risk problem, and we show that the worst-case risk minimization problem admits a tractable convex reformulation due to the regularity of the CS loss. Moreover, we develop a kernel version of our proposed model to account for nonlinear class separation, and we show that it admits a tractable convex upper bound. We also propose a projected subgradient method algorithm for a special case of our proposed linear model to improve scalability. Our numerical experiments demonstrate that our model outperforms state-of-the art OVA models in settings where the training data is highly imbalanced. We also show through experiments on popular real-world datasets that our proposed model often outperforms its regularized counterpart as the first accounts for uncertain labels unlike the latter.