Distributionally Robust Logistic Regression

TL;DR

This paper introduces a distributionally robust logistic regression framework using Wasserstein distances, providing guarantees on out-of-sample performance and encompassing classical and regularized models.

Contribution

It formulates a tractable distributionally robust logistic regression model with high-confidence guarantees, unifying classical and regularized approaches.

Findings

Provides a tractable reformulation of the robust optimization problem.

Offers confidence bounds on misclassification probability via linear programs.

Validates theoretical guarantees through simulations and empirical data.

Abstract

This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.