Distributionally Robust Multiclass Classification and Applications in Deep Image Classifiers

TL;DR

This paper introduces a distributionally robust optimization framework for multiclass logistic regression that enhances robustness against outliers and adversarial attacks, with demonstrated significant improvements in deep image classifiers on MNIST and CIFAR-10 datasets.

Contribution

The paper develops a DRO-based approach for multiclass logistic regression, providing theoretical guarantees and applying it to improve robustness of deep vision transformers against attacks.

Findings

Up to 83.5% reduction in test error rate on MNIST and CIFAR-10.

Up to 91.3% decrease in loss compared to baseline methods.

Effective in defending deep classifiers from random and adversarial attacks.

Abstract

We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR), which could tolerate data contaminated by outliers. The DRO framework uses a probabilistic ambiguity set defined as a ball of distributions that are close to the empirical distribution of the training set in the sense of the Wasserstein metric. We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix. We establish out-of-sample performance guarantees for the solutions to our model, offering insights on the role of the regularizer in controlling the prediction error. We apply the proposed method in rendering deep Vision Transformer (ViT)-based image classifiers robust to random and adversarial attacks. Specifically, using the MNIST and CIFAR-10 datasets, we demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.