General risk measures for robust machine learning

TL;DR

This paper introduces a novel framework for robust machine learning by applying risk measures from quantitative finance, transforming min-max problems into convex optimization problems with practical algorithms and demonstrating their effectiveness on real data.

Contribution

It adapts risk measure frameworks to machine learning, providing convex reformulations and efficient algorithms for robust models under distributional uncertainty.

Findings

Convex reformulation of min-max problems under certain assumptions

Development of an efficient algorithm for complex convex constraints

Successful scaling of the algorithm on real datasets

Abstract

A wide array of machine learning problems are formulated as the minimization of the expectation of a convex loss function on some parameter space. Since the probability distribution of the data of interest is usually unknown, it is is often estimated from training sets, which may lead to poor out-of-sample performance. In this work, we bring new insights in this problem by using the framework which has been developed in quantitative finance for risk measures. We show that the original min-max problem can be recast as a convex minimization problem under suitable assumptions. We discuss several important examples of robust formulations, in particular by defining ambiguity sets based on $\varphi$-divergences and the Wasserstein metric.We also propose an efficient algorithm for solving the corresponding convex optimization problems involving complex convex constraints. Through simulation examples, we demonstrate that this algorithm scales well on real data sets.

Figures

Tables

Table 1. Table 1 : Parameters of the datasets.

Name of dataset	ionosphere	colon-cancer
Number of observations ($N$)	351	64
Number of features ($d$)	34	2000

Table 2. Table 2 : colon-cancer dataset: Values of the AUC for different values of ϵ italic-ϵ \epsilon .

Value of $ϵ$	AUC with KL	AUC with Wasserstein
$ϵ=0$ (LR)	0.832	0.832
$ϵ=0.001$	0.757	0.787
$ϵ=0.002$	0.750	0.770
$ϵ=0.003$	0.779	0.706
$ϵ=0.004$	0.698	0.691
$ϵ=0.005$	0.868	0.831
$ϵ=0.006$	0.890	0.860
$ϵ=0.007$	0.728	0.838
$ϵ=0.008$	0.809	0.768
$ϵ=0.009$	0.875	0.890
$ϵ=0.01$	0.801	0.853
$ϵ=0.05$	0.786	0.794
$ϵ=0.1$	0.801	0.816

Table 3. Table 3 : ionosphere dataset (altered): Values of the area under ROC curve for different values of ϵ italic-ϵ \epsilon .

Value of $ϵ$	AUC with KL	AUC with Wasserstein
$ϵ=0$ (LR)	0.514	0.514
$ϵ=0.001$	0.816	0.840
$ϵ=0.002$	0.804	0.835
$ϵ=0.003$	0.840	0.814
$ϵ=0.004$	0.824	0.830
$ϵ=0.005$	0.815	0.829
$ϵ=0.006$	0.834	0.829
$ϵ=0.007$	0.821	0.815
$ϵ=0.008$	0.835	0.815
$ϵ=0.009$	0.823	0.822
$ϵ=0.01$	0.828	0.835
$ϵ=0.05$	0.815	0.826
$ϵ=0.1$	0.824	0.823

Equations

$$\bigcap_{i=1}^{N}\mathop{\mathrm{dom}}\big{(}\ell(\cdot,z_{i})\big{)}\neq\varnothing\;,$$

$$\overline{\theta }\in {\mathrm{argmin}}_{\theta \in {\mathbb{R}}^n}\sum _{i=1}^N{p}_i\ell (\theta ,z_i).$$

$$(\forall i\in [[1,N]])(\forall \theta \in {\mathbb{R}}^n)\ell (\theta ,z_i)=\frac{1}{2}\Vert y_i-x_i^{\top }\theta {\Vert }^2+\rho (\theta ),$$

$$(\forall i\in[\![1,N]\!])(\forall\theta\in\mathbb{R}^{n})\quad\ell(\theta,z_{i})=\log\Big{(}1+\exp\big{(}-y_{i}x_{i}^{\top}\theta\big{)}\Big{)}\;.$$

$$\overline{\theta }\in {\mathrm{argmin}}_{\theta \in {\mathbb{R}}^n}\underset{q=(q_i{)}_{1\le i\le N}\in {\mathcal{M}}_1}{\sup }\left(\sum _{i=1}^N{q}_i\ell (\theta ,z_i)-\alpha (q)\right),$$

$$\sup_{q\in\mathcal{Q}}\sum_{i=1}^{N}q_{i}\ell(\theta,z_{i})=\sigma_{\mathcal{Q}}\big{(}\mathbf{\ell}(\theta,\mathbf{z})\big{)}\;,$$

$$(\forall ({\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}},{\mathbf{Y}}_{\text{unboldmath}}^{\text{unboldmath}})\in {\mathbb{X}}^2){\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}\le {\mathbf{Y}}_{\text{unboldmath}}^{\text{unboldmath}}\iff (\forall \omega \in \Omega ){\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}(\omega )\le {\mathbf{Y}}_{\text{unboldmath}}^{\text{unboldmath}}(\omega ).$$

$$(\forall {\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}\in \mathbb{X})\mathbb{F}[{\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}]=\underset{s\in \mathbb{R}}{\inf }\{s\mid {\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}-s\in {\mathrm{lev}}_{\le 0}\mathbb{F}\},$$

$$\operatorname{lev}_{\leq 0}\mathbb{F}=\big{\{}\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}\in\mathbb{X}\mid\mathbb{F}\big{[}\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}\big{]}\leq 0\big{\}}\;.$$

$$(\forall {\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}\in \mathbb{X})\mathbb{F}[{\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}]=\underset{q\in {\mathcal{M}}_1}{\sup }\left(\sum _{i=1}^N{q}_i{x}_i-\alpha (q)\right).$$

$$(\forall q\in\mathbb{R}^{N})\quad\alpha(q)=\begin{cases}\displaystyle\sup_{\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}\in\operatorname{lev}_{\leq 0}\mathbb{F}}\mathbb{E}_{q}[\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}]&\mbox{if $q\in\mathcal{M}_{1}\;,$}\\ +\infty&\mbox{otherwise}\;.\end{cases}$$

$$(\forall\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}\in\mathbb{X})\quad\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}]=\sup_{q\in\mathcal{M}_{1}}\big{(}\mathbb{E}_{q}[\boldsymbol{X^{\textstyle\text{\unboldmath$\scriptstyle$}}_{\textstyle\text{\unboldmath$\scriptstyle$}}}]-\alpha(q)\big{)}\;,$$

$$(\forall q\in {\mathcal{M}}_1)\alpha (q)$$

$$=\underset{{\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}\in {\mathrm{lev}}_{\le 0}\mathbb{F}}{\sup }{\mathbb{E}}_q[{\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}]$$

$$\overline{\theta}\in\mathop{\arg\min}_{\theta\in\mathbb{R}^{n}}\mathbb{F}_{\alpha}\big{[}\ell(\theta,\mathbf{z})\big{]}\;,$$

$$(\forall {\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}\in \mathbb{X}){\mathbb{F}}_{\alpha }[{\mathbf{X}}_{\text{unboldmath}}^{\text{unboldmath}}]=\underset{q\in {\mathcal{M}}_1}{\max }\left(\sum _{i=1}^N{q}_i{x}_i-\alpha (q)\right).$$

$$\underset{(\theta ,s)\in \mathcal{S}}{\inf }s$$

$$\mathcal{S}=\{(\theta ,s)\in {\mathbb{R}}^n\times \mathbb{R}\mid \ell (\theta ,\mathbf{z})-s\in {\mathrm{lev}}_{\le 0}{\mathbb{F}}_{\alpha }\}$$

$$(\forall \omega \in \Omega )\ell (\lambda {\theta }_1+(1-\lambda ){\theta }_2,\mathbf{z}(\omega ))\le \lambda \ell ({\theta }_1,\mathbf{z}(\omega ))+(1-\lambda )\ell ({\theta }_2,\mathbf{z}(\omega )).$$

$$\displaystyle\mathbb{F}_{\alpha}\Big{[}\ell(\lambda\theta_{1}+(1-\lambda)\theta_{2},\mathbf{z})\Big{]}$$

$$\displaystyle\leq\lambda\mathbb{F}_{\alpha}\Big{[}\ell(\theta_{1},\mathbf{z})\Big{]}+(1-\lambda)\mathbb{F}_{\alpha}\Big{[}\ell(\theta_{2},\mathbf{z})\Big{]}\;.$$

$$f_{\phi }:\mathbb{R}\times \mathbb{R}$$

$$(x,\xi )$$

$$\big{(}\forall p=(p_{i})_{1\leq i\leq N}\in\mathbb{R}^{N}\big{)}\big{(}\forall q=(q_{i})_{1\leq i\leq N}\in\mathbb{R}^{N}\big{)}\quad D_{\varphi}(p,q)=\sum_{i=1}^{N}\underline{f_{\varphi}}(p_{i},q_{i})\;,$$

$$\underline{f_{\phi }}:\mathbb{R}\times \mathbb{R}$$

$$(x,\xi )$$

$$(\forall s\in\mathbb{R})\quad\varphi^{*}(s)=\sup_{t\in\mathbb{R}}\big{(}st-\varphi(t)\big{)}\;,$$

$$(\forall t~\in\mathbb{R})\quad\tilde{\varphi}(t)=\begin{cases}\displaystyle t\varphi\left(\frac{1}{t}\right)&\mbox{if $t\geq 0$}\;,\\ \displaystyle\lim_{t\to+\infty}\frac{\varphi(t)}{t}&\mbox{if $t=0$\; .}\end{cases}$$

$$(\forall (p,q)\in ({\mathbb{R}}^N{)}^2)D_{\tilde{\phi }}(p,q)=D_{\phi }(q,p)$$

$$(\forall s\in \mathbb{R})(\lambda \phi {)}^{\ast }(s)=\lambda {\phi }^{\ast }\left(\frac{s}{\lambda }\right).$$

$$(\forall q\in\mathbb{R}^{N})\quad\alpha(q)=\begin{cases}\lambda_{0}D_{\varphi}(q,p)&\mbox{if $q\in\mathcal{M}_{1}$}\;,\\ +\infty&\mbox{otherwise}\;,\end{cases}$$

$$\overline{\theta }={\mathrm{argmin}}_{\theta \in {\mathbb{R}}^n}\underset{\mu \in \mathbb{R}}{\min }g(\theta ,\mu ),$$

Full text

Keywords: Risk measures, robust statistics, machine learning, convex optimization, divergences, Wasserstein distance.**

1 Introduction

In machine learning, the robustness of the solutions obtained for classification and prediction tasks remains a main issue. In Papernot, McDaniel, and Goodfellow (2016)* and Kurakin, Goodfellow, and Bengio (2016), some examples are provided where small modifications of the input data can completely alter the resulting solution. In Feng, Xu, Mannor, and Yan (2014) and Plan and Vershynin (2013), poor out-of-sample performances are displayed when training data is sparse. This kind of problems also occurs in optimal control when there exist uncertainties on parameters. In Ben-Tal and Nemirovski (2000), the authors showed that a small perturbation on the parameters can turn a feasible solution into an infeasible one.*

In this context, robust approaches appear as a way of controlling out-of-sample performance. There is an extensive literature dealing with robust problems and the reader is refered to Ben-Tal, El Ghaoui, and Nemirovski (2009)* for a survey. One of the main approaches consists of introducing constraints on the probability distribution of the unknown data. Under some conditions, this approach is equivalent to deal with ambiguity sets or a modified loss function. The works in Ben-Tal, Den Hertog, De Waegenaere, Melenberg, and Rennen (2013); Hu and Hong (2013); Duchi, Glynn, and Namkoong (2016); Moghaddam and Mahlooji (2016)** and Namkoong and Duchi (2016)** have brought more insight on ambiguity sets. In Esfahani and Kuhn (2015)** and Esfahani, Shafieezadeh-Abadeh, Hanasusanto, and Kuhn (2017)**, the authors present a distributionally robust optimization framework based on the Wasserstein distance. A set of probability distributions is defined as a ball centered on the reference probability with respect to the Wasserstein distance, then the optimization is carried out for the worst cost over this probability set.*

This idea of minimizing the worst cost over a given probability set is well-known in quantitative finance. The robust representation of risk measures provides a theoretical framework to do so. A rich class of risk measures is the class of coherent ones which were introduced in the seminal paper by Artzner, Delbaen, Eber, and Heath (1999). In Föllmer and Schied (2016), a broader class of so-called convex risk measures was investigated, for which a large number of results were established.

In this paper, we follow the line of Esfahani and Kuhn (2015), which aims at reformulating robust problems using ambiguity sets as convex minimization problems. Our contribution is threefold. First we clarify the links existing between risk measures and robust optimization. This allows us to transpose results from finance to machine learning. Second, we propose a unifying convex optimization setting for dealing with various risk measures, including those based on $\varphi$-divergences or the Wasserstein distance. Finally, we propose an accelerated algorithm grounded on the subgradient projection method proposed in Combettes (2003). We show that the proposed algorithm is able to solve efficiently large-scale robust problems.

The organization of the paper is as follows. In Section 2, we state the general mathematical problem we investigate in the context of machine learning. In Section 3, we first draw a parallel between this problem and convex monetary risk measures. We then provide a convex reformulation of the problem. In Section 4, we discuss some important classes of risk measures by revisiting some of the results in the literature. In Section 5, we describe our algorithm for solving convex formulations of robust problems. Then, in Section 6, we illustrate the good performance of the proposed algorithm through numerical experiments on real datasets. Finally, short concluding remarks are made in Section 7.

2 Problem statement

Let $(\Omega,\mathscr{F},p)$ be the underlying probability space where $\Omega$ is a finite set of cardinal $N$, $\mathscr{F}$ is the $\sigma$-field generated by $(\{\omega\})_{\omega\in\Omega}$, and $p$ is a probability distribution that is assumed to charge all points. Let $d$ be a nonzero integer and let $\mathbf{z}\colon\Omega\to\mathbb{R}^{d}$ denote a general random variable. Note that function $\mathbf{z}$ can be identified with a matrix in $\mathbb{R}^{N\times d}$ where, for every $i\in[\![1,N]\!]$, $z_{i}\in\mathbb{R}^{d}$ is the vector corresponding to the $i$-th line of matrix $\mathbf{z}$ . We denote by $\mathcal{M}_{1}$ the set of probability distributions over $(\Omega,\mathscr{F},p)$.

For every $i\in[\![1,N]\!]$, let $\ell(\cdot,z_{i})\colon\mathbb{R}^{n}\to\mathbb{R}\cup\{+\infty\}$ be a loss function which is assumed to be lower semicontinuous (lsc) and convex such that

[TABLE]

where $\mathop{\mathrm{dom}}(g)$ denotes the domain of a function $g$, that is the set of argument values for which this function is finite. In standard formulations of machine learning problems, one aims at finding an optimal regression vector $\overline{\theta}\in\mathbb{R}^{n}$ such that

[TABLE]

Indeed, setting $z_{i}=[x_{i}^{\top}\;\;y_{i}]^{\top}$ with $n=d-1\geq 1$, $x_{i}\in\mathbb{R}^{n}$, and $y_{i}\in\mathbb{R}$ allows us to recover a wide array of estimation and classification problems. For example, penalized least squares regression problems are obtained when

[TABLE]

where $\rho\colon\mathbb{R}^{n}\to\mathbb{R}\cup\{+\infty\}$ is a proper, lsc, convex penalty function. If the random variable $\mathbf{y}$ is $\{0,1\}$-valued, we can recover binary classification problems, for example by performing a logistic regression, i.e.

[TABLE]

One of the main limitations of this formulation is that it assumes that the true probability distribution of the data is perfectly known. In practice, this distribution is often estimated empirically from the available observations.

In this paper, we will focus on the following more general robust formulation to determine an optimal regression vector.

Problem 1**.**

Let $\alpha\colon\mathbb{R}^{N}\to\mathbb{R}\cup\{+\infty\}$ be a lsc convex penalty function whose domain is a nonempty subset of $\mathcal{M}_{1}$. We want to find

[TABLE]

In this problem, if $\alpha$ is the indicator function $\iota_{\{p\}}$ of the singleton containing a probability distribution $p$, then (2) is recovered.111The indicator of a set is the function equal to 0 on this set and $+\infty$ otherwise. More generally, if $\alpha$ is equal to the indicator function of a nonempty closed convex set $\mathcal{Q}\subset\mathcal{M}_{1}$, then the objective function in (5) reduces to

[TABLE]

where $\sigma_{\mathcal{Q}}$ is the support function of $\mathcal{Q}$. This corresponds to the well-known case of distributionally robust optimization using ambiguity sets Esfahani and Kuhn (2015)**.

3 Convex formulation of robust inference problems using risk measures

In this section, we address Problem 1 in the light of the financial framework for monetary risk measures. We first recall known properties of risk measures and then show how Problem 1 can be reformulated as a convex problem.

3.1 Definition and properties of a risk measure

Let $\mathbb{X}$ be the space of real-valued random variables defined on the probability space $(\Omega,\mathscr{F},p)$. We denote by \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}} a generic element of $\mathbb{X}$ and we recall that $p$ is assumed to be a distribution that charges all points. The space $\mathbb{X}$ is endowed with the pointwise order $\leq$, that is,

[TABLE]

*A risk measure $\mathbb{F}$ is a real-valued function *$\mathbb{F}:\mathbb{X}\to\mathbb{R}\;.$

The next four properties of risk measures were first introduced in Artzner, Delbaen, Eber, and Heath (1999)* to define the so called coherent risk measures. The interested reader can also refer to Föllmer and Schied (2016)**[Part I, Chapter 4].*

Definition 1**.**

A risk measure $\mathbb{F}\colon\mathbb{X}\to\mathbb{R}$ is said to be

• •

monotone: if, for every (\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}},\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}})\in\mathbb{X}^{2}, \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\leq\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\Rightarrow\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]\leq\mathbb{F}[\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}],

• •

translation invariant: if, for every \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\in\mathbb{X} and $m\in\mathbb{R}$, \mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}+m]=\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]+m,

• •

convex: if, for every (\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}},\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}})\in\mathbb{X}^{2} and $\lambda\in]0,1[$, \mathbb{F}\big{[}\lambda\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}+(1-\lambda)\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\big{]}\leq\lambda\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]+(1-\lambda)\mathbb{F}[\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}],

• •

positively homogeneous: if, for every \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\in\mathbb{X} and $\lambda\in[0,+\infty[$, \mathbb{F}[\lambda\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]=\lambda\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}].

A risk measure which satisfies the first two properties is called a monetary risk measure. A risk measure which satisfies the first three properties is called a convex risk measure. A risk measure which satisfies the four properties is called a coherent risk measure.

Depending on the author, the first axiom may also be expressed as: for every (\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}},\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}})\in\mathbb{X}^{2}, \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\leq\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\Rightarrow\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]\geq\mathbb{F}[\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}] if the variables \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}} and \boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}} are interpreted as gains instead of a losses, which is a common in finance. For this reason, some sign differences may appear between results of various authors. We have chosen to follow the paths in Rockafellar and Uryasev (2000); Ruszczynski and Shapiro (2006); Ruszczyński and Shapiro (2006)** and interpret the random variable in argument as a loss. We however often refer to Föllmer and Schied (2016)**, providing a comprehensive view of risk measures, where the opposite convention has been adopted.

Remark 2*.*

1. (i)

It readily follows from the translation invariance property that a monetary risk measure $\mathbb{F}$ admits a primal form given by

[TABLE]

where $\operatorname{lev}_{\leq 0}\mathbb{F}$ is the lower level set of $\mathbb{F}$ at height 0 defined as

[TABLE] 2. (ii)

A monetary risk measure $\mathbb{F}$ is $1$-Lipschitz continuous with respect to the supremum norm $\|\cdot\|_{\infty}$. Indeed, for every (\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}},\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}})\in\mathbb{X}^{2}, we have \boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\leq\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}+\|\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}-\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\|. By monotonicity and translation invariance we obtain that \mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]-\mathbb{F}[\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]\leq\|\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}-\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\|_{\infty}, which by symmetry implies that |\mathbb{F}[\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]-\mathbb{F}[\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}]|\leq\|\boldsymbol{X^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}-\boldsymbol{Y^{\textstyle\text{\unboldmath\scriptstyle}}_{\textstyle\text{\unboldmath\scriptstyle}}}\|_{\infty}.

The class of convex risk measures includes a large number of useful functions. Without entering into details, we should mention: expectation, worst case, quantile, median, and average value at risk Föllmer and Schied (2016)**.

3.2 Convex reformulation

In this section, we will show that the “min-max” problem 1 admits a convex reformulation. We first gather in the following proposition some existing results in the literature.

Proposition 3**.**

$\mathbb{F}{}$* is a convex risk measure if and only if there exists a lsc and convex function $\alpha\colon\mathbb{R}^{N}\to\mathbb{R}\cup\{+\infty\}$ such that*

[TABLE]

The function $\alpha$ associated with $\mathbb{F}{}$ is uniquely defined as

[TABLE]

In addition, $\mathbb{F}{}$ is coherent if and only if its conjugate function $\alpha$ is the indicator function of a nonempty closed convex subset of $\mathcal{M}_{1}$.

Proof.

1. (i)

We know from Föllmer and Schied (2016, Theorem 4.16 and Proposition 4.15) that any convex risk measure $\mathbb{F}$ on $\mathbb{X}$ is of the form

[TABLE]

where $\alpha\colon\mathbb{R}^{N}\to\mathbb{R}\cup\{+\infty\}$ is the lsc and convex function whose domain is a nonempty subset of $\mathcal{M}_{1}$, given by

[TABLE]

(the second equality stems from Remark 2(i)).

Conversely, one can associate to every lsc convex function $\alpha\colon\mathbb{R}^{N}\to\mathbb{R}\cup\{+\infty\}$ whose domain is a nonempty subset of $\mathcal{M}_{1}$ a unique convex risk measure defined by (12). 2. (ii)

It follows from Föllmer and Schied (2016, Proposition 4.15) that if, in addition, the risk measure $\mathbb{F}$ is coherent, then the function $\alpha$ in (11) is the indicator function of a nonempty closed convex subset of $\mathcal{M}_{1}$ and the converse property holds.

∎

We now state the main result of this section.

Theorem 4**.**

Let $\alpha\colon\mathbb{R}^{N}\to\mathbb{R}\cup\{+\infty\}$ be a lsc convex function whose domain is a nonempty subset of $\mathcal{M}_{1}$. Problem 1 is equivalent to find

[TABLE]

where

[TABLE]

The function \mathbb{F}_{\alpha}\big{[}\ell(\cdot,\mathbf{z})\big{]} is proper, lsc, and convex. In addition, the so-defined convex optimization problem admits a primal formulation which consists of finding

[TABLE]

where

[TABLE]

.

Proof.

It follows from Proposition 3 that (5) is equivalent to (14) where $\mathbb{F}_{\alpha}$ is a convex risk measure. In addition, the $\sup$ in the definition of the risk measure is attained since $\mathcal{M}_{1}$ is a compact set and $q\mapsto\sum_{i=1}^{N}q_{i}x_{i}-\alpha(q)$ is upper semicontinuous.

The function $\ell(\cdot,Z)$ is lsc convex for every $Z\in\mathbb{R}^{d}$. Given a random variable $\mathbf{z}$, for every vectors $\theta_{1}$ and $\theta_{2}$ in $\mathbb{R}^{n}$, and scalar $\lambda\in[0,1]$, the convexity of function $\ell$ yields

[TABLE]

Now, by using the fact that the risk measure $\mathbb{F}_{\alpha}$ is monotone and convex with respect to the ordering introduced in (7), we get

[TABLE]

This shows that \mathbb{F}_{\alpha}\big{[}\ell(\cdot,\mathbf{z})\big{]} is convex.

In addition, since $\mathbb{F}_{\alpha}{}$ is monotone and continuous (see Remark 2(ii)) and $\ell(\cdot,\mathbf{z})$ is lsc, \mathbb{F}_{\alpha}\big{[}\ell(\cdot,\mathbf{z})\big{]} is lsc. Because of (1), \mathbb{F}_{\alpha}\big{[}\ell(\cdot,\mathbf{z})\big{]} is also proper.

Finally, formulation (16) is deduced from (8) and (14).

∎

The general convex reformulation (16) is not always easy to handle. In practical applications, the choice of the mapping $\alpha$ plays a crucial role in this regard. We will see in the next section some useful examples of this function. In particular, some mappings $\alpha$ lead to a formulation (16) that will be shown to be tractable numerically.

4 Examples of risks measures

By considering particular forms of function $\alpha$ in Problem 1, we define three scenarios of interest for robust formulations. The first two ones are based on $\varphi$-divergences, while the third one is based on the Wasserstein metric.

4.1 Perspective functions and divergences

The notion of $\varphi$-divergence was first introduced independently by Csiszár (1964); Morimoto (1963)** and Ali and Silvey (1966). For a more complete bibliography on the subject, we refer to Basseville (2013).

Definition 5**.**

Let $\varphi:\mathbb{R}\to]-\infty,+\infty]$. The perspective function $f_{\varphi}$ of function $\varphi$ is given by

[TABLE]

Definition 6**.**

Let $\varphi:\mathbb{R}\to[0,+\infty]$ be a lsc convex function with nonempty domain included in $[0,+\infty[$ such that $\varphi(1)=0$. The $\varphi$-divergence $D_{\varphi}\colon\mathbb{R}^{N}\times\mathbb{R}^{N}\to[0,+\infty]$ is defined as

[TABLE]

where the function $\underline{f_{\varphi}}$ is the lsc envelope of the mapping $f_{\varphi}$, that is

[TABLE]

We also recall the definitions of a conjugate function and an adjoint function.

Definition 7**.**

Let $\varphi:\mathbb{R}\to]-\infty,+\infty]$. The conjugate $\varphi^{*}$ of function $\varphi$ is defined by

[TABLE]

and the so-called adjoint function of $\varphi$ is defined by

[TABLE]

Table LABEL:table_phi_divergence is an extension of the one in Ben-Tal, Den Hertog, De Waegenaere, Melenberg, and Rennen (2013)* and provides the expressions of common $\varphi$ functions, their conjugates, and the associated $\varphi$-divergence. It is well-known (Ben-Tal, Den Hertog, De Waegenaere, Melenberg, and Rennen, 2013; Combettes and Müller, 2018) that the adjoint $\tilde{\varphi}$ of $\varphi$ is such that*

[TABLE]

and the conjugate of function $\lambda\varphi$ is

[TABLE]

4.2 Divergence penalty functions

A first case of interest is when the penalty term $\alpha(q)$ in Problem 1 measures the “distance” between $p$ and $q$ in the sense of a $\varphi$-divergence.

Proposition 8**.**

Let $\varphi:\mathbb{R}\to[0,+\infty]$ be a lsc convex function with nonempty domain included in $[0,+\infty[$, which is such that $\varphi(1)=0$. Let $\alpha$ be the function given by

[TABLE]

with $\lambda_{0}\in]0,+\infty[$. Problem 1 is equivalent to find

[TABLE]

where $g$ is the proper, lsc, convex function given by

[TABLE]

Proof.

We can reexpress (5) as

[TABLE]

It follows from Föllmer and Schied (2016, Theorem 4.122) that

[TABLE]

The equivalence between (37) and (35) then results from Theorem 4.

In addition, plugging the expression of $\varphi^{*}$ in (36) yields

[TABLE]

For every $(t_{i})_{i\in[\![1,N]\!]}\in\mathbb{R}^{N}$,

[TABLE]

is a lsc convex function. Since convexity and lower semicontinuity are kept by the supremum operation, $g$ is lsc and convex. By using (1), (36), and the fact that $\varphi^{*}$ is proper, there exist $\theta\in\mathbb{R}^{n}$ and $\mu\in\mathbb{R}$, such that $g(\theta,\mu)<+\infty$.

∎

4.3 Constrained formulations

We now investigate two particular cases when $\alpha$ is the indicator function of a convex set $\mathcal{Q}$ of probability distributions, so defining an ambiguity set.

4.3.1 Ball with respect to a divergence

A first possibility is to introduce an upper bound on the divergence $D_{\varphi}(q,p)$ between the sought distribution $q$ and $p$ by considering the constraint set

[TABLE]

where $\epsilon\in]0,+\infty[$.

The following result generalizes both Ben-Tal et al. (2013)* where the authors deal with linear costs under constraints and Hu and Hong (2013)** where the authors focus on the Kullback-Leibler divergence.*

Proposition 9**.**

Let $\varphi:\mathbb{R}\to[0,+\infty]$ be a lsc convex function such that $\mathop{\mathrm{dom}}(\varphi)=]0,+\infty[$ or $\mathop{\mathrm{dom}}(\varphi)=[0,+\infty[$, and $\varphi(1)=0$. Let $\epsilon\in]0,+\infty[$ and let $\alpha=\iota_{\mathbb{B}_{\epsilon}^{\varphi}}$. Problem 1 is equivalent to find

[TABLE]

where $g$ is the proper, lsc, convex function given by

[TABLE]

with the convention

[TABLE]

Proof.

The risk function associated with $\alpha=\iota_{\mathbb{B}_{\epsilon}^{\varphi}}$ is

[TABLE]

Since $1$ belongs to the interior of $\mathop{\mathrm{dom}}(\varphi)$ and

[TABLE]

Slater’s condition holds for constraint (45b). Since the constraint is feasible and $q\mapsto-\sum_{i=1}^{N}q_{i}x_{i}+\iota_{\mathcal{M}_{1}}(q)$ is lsc, convex, and coercive, there exists a solution $\overline{q}\in\mathcal{M}_{1}$ to the above constrained maximization problem. It then follows from standard Lagrange duality for convex functions that there exists $\overline{\lambda}\in[0,+\infty[$ such that $(\overline{q},\overline{\lambda})$ is a saddle point of the Lagrange function

[TABLE]

where

[TABLE]

We have thus

[TABLE]

where, for every $\lambda\in[0,+\infty[$,

[TABLE]

Two cases will be distinguished.

1. (i)

Case when $\lambda=0$.

Then (50) reduces to

[TABLE]

where

[TABLE]

In addition, since $]0,+\infty[\in\mathop{\mathrm{dom}}(\varphi)$, the upper bound in (51) is attained, yielding

[TABLE] 2. (ii)

Case when $\lambda>0$.

(50) can be reexpressed as

[TABLE]

where

[TABLE]

The conjugate of $\Phi$ reads

[TABLE]

whereas the conjugate of $\iota_{\mathcal{M}_{1}}$ is given by $\sigma_{\mathcal{M}_{1}}$ in (52). Since $\sigma_{\mathcal{M}_{1}}$ is finite valued, the conjugate of $\Phi+\iota_{\mathcal{M}_{1}}$ is given by the following inf-convolution (Bauschke and Combettes, 2011, Theorem 15.3)

[TABLE]

which, by using ((ii)), yields

[TABLE]

Since $\mathop{\mathrm{dom}}(\lambda\varphi)\subset[0,+\infty[$, $(\lambda\varphi)^{*}\colon\xi\mapsto\sup_{\upsilon\in[0,+\infty[}\xi\upsilon-\lambda\varphi(\upsilon)$ is an increasing function. This implies that

[TABLE]

Note that the right-hand side in the previous formula when applied at $\lambda=0$ by using (44) and (52) reduces to

[TABLE]

Consequently, (49) leads to

[TABLE]

and (42) follows from Theorem 4.

In addition, by using the expression of the conjuguate, $g$ can be reexpressed as

[TABLE]

As a supremum of lsc convex functions, $g$ also is lsc convex. The fact that $g$ is proper follows from arguments similar to those at the end of the proof of Proposition 8.

∎

Remark 10*.*

The divergence risk measure in (38) is convex, whereas the risk measure in (61) is coherent (see Proposition 3), which means that the risk scales with the data in the latter case.

4.3.2 Ball with respect to the Wasserstein metric

We now investigate Problem 1 when function $\alpha$ is the indicator of a Wasserstein ball centered on $p$. For this purpose, we first recall the notion of Wasserstein distance.

Definition 11**.**

Let $\mathcal{M}(\Xi^{2})$ denote the set of probability distributions supported on $\Xi^{2}$. The Wasserstein distance between two distributions $p$ and $q$ supported on $\Xi$ is defined as

[TABLE]

where $\delta$ is a metric on $\Xi$.

We now introduce the notion of Wasserstein ball. The considered constrained set is denoted by

[TABLE]

with $\epsilon\in]0,+\infty[$.

In the following theorem, $\delta$ is the usual Euclidean distance. The following convex reformulation of Problem 1 can be derived from (Esfahani and Kuhn, 2015, Theorem 4.2).

Proposition 12**.**

Let $\epsilon\in]0,+\infty[$ and let $\alpha=\iota_{\mathbb{B}_{\epsilon}^{\mathbb{W}}}$. Then, Problem 1 is equivalent to find

[TABLE]

where $g$ is the proper, lsc convex function given by

[TABLE]

where $\mathcal{W}$ is the closed convex set defined as

[TABLE]

5 Numerical solution

We will now propose an algorithm allowing us to solve numerically the three convex optimization problems in Propositions 8, 9, and 12. This algorithm applies to more general choices of function $\alpha$ in Problem 1 where the constraint $\mathcal{S}$ in (17) splits as an intersection of a finite number of convex constraints.

5.1 A unifying formulation

We first show that the convex optimization problems discussed in Section 4 can be reexpressed in a unifying manner.

Proposition 13**.**

The optimization problems in Propositions 8, 9, and 12 amount to finding

[TABLE]

where $K\in\mathbb{N}\setminus\{0\}$ and the functions \big{(}f_{k}\big{(}\cdot,\mathbf{z}\big{)}\big{)}_{k\in[\![1,K]\!]} are proper, lsc, and convex. More precisely,

1. (i)

for divergence penalty functions, $K=N$ and, for every $k\in[\![1,K]\!]$,

[TABLE] 2. (ii)

for divergence ball constraints, $K=N$ and, for every $k\in[\![1,K]\!]$,

[TABLE] 3. (iii)

for the Wassertein ball constraint, $K=N^{2}$ and, for every $k\in[\![1,K]\!]$ and $(i_{k},j_{k})\in[\![1,N]\!]^{2}$ such that $k=N(i_{k}-1)+j_{k}$,

[TABLE]

5.2 Description of the algorithm

In this section, we propose an accelerated projected gradient algorithm for solving Problem (68). One step of this proximal algorithm Combettes and Pesquet (2010)* reads as a projection onto a set defined as an intersection of non trivial closed convex sets. To solve this projection problem, we use the subgradient projection algorithm in Combettes (2003), which is related to ideas introduced in Haugazeau (1968, Theorem 3-2). This algorithm allows the constraints to be activated individually in a flexible parallel manner. We will first recall the basic structure of our algorithm before describing in more details the subgradient projection step. *

Proximal algorithm

Let $\mathcal{H}=\mathbb{R}^{n}\times\mathbb{R}\times\mathbb{R}\times\mathbb{R}^{N}$ and let $\|\cdot\|$ (resp. $\left\langle\cdot\>,\cdot\right\rangle$) denote the standard norm (resp. the inner product) equiping this product space. By introducing the generic variable $u=(\theta,\lambda,\mu,s)\in\mathcal{H}$, (68) can be reexpressed more concisely as

[TABLE]

where $c=(0,\epsilon,1,p)\in\mathcal{H}$ and $\mathcal{C}=\cap_{k=0}^{K}\mathcal{C}_{k}$ with

[TABLE]

To solve the above problem, we propose to employ a FISTA-like algorithm Beck and Teboulle (2009)**. Let $n\in\mathbb{N}\setminus\{0\}$. The $n$-th iteration of this algorithm reads

[TABLE]

where $\gamma\in]0,+\infty[$ and $\operatorname{P}_{\mathcal{C}}\colon\mathcal{H}\to\mathcal{C}$ is the projection onto the closed convex set $\mathcal{C}$. It follows from (Chambolle and Dossal, 2015, Theorem 3) that, if a solution to the minimization problem exists, and

[TABLE]

then the convergence of $(u^{(n)})_{n\in\mathbb{N}}$ to a solution to the problem is guaranteed.

The main difficulty in the implementation of the algorithm lies in the computation of the projection onto $\mathcal{C}$ that will be discussed next.

Computation of the projection

Algorithm 1 presents our projection method inspired from Combettes (2003)**. At iteration $\ell\in\mathbb{N}$, $Q(p^{(0)},p^{(\ell)},r^{(\ell)})$ designates the projection of $p^{(0)}$ onto the intersection of the 3 half-spaces $\mathcal{C}_{0}$, $H_{\ell}$, and $D_{\ell}$, where

[TABLE]

Since the projection onto $H_{\ell}\cap D_{\ell}$ has an explicit form Combettes (2003), a dual forward-backward algorithm Combettes et al. (2010) allows us to compute in a fast manner the projection onto $\mathcal{C}_{0}\cap H_{\ell}\cap D_{\ell}$. The algorithm has been intialized by setting $p^{(0)}=\operatorname{P}_{\mathcal{C}_{0}}(v^{(n)}-\gamma c)$, taking into account the fact that $\operatorname{P}_{\mathcal{C}}=\operatorname{P}_{\mathcal{C}}\circ\operatorname{P}_{\mathcal{C}_{0}}$. At each iteration $\ell$, $\mathbb{K}_{\ell}$ designates the set of indices of the constraints which are activated. When dealing with large-scale problems, it may be useful not to require all the constraints to be activated at each iteration. The convergence of the algorithm is guaranteed by the study in Combettes (2000)**, provided that, for every $k\in[\![1,K]\!]$, $\mathcal{C}_{0}\subset\mathop{\mathrm{dom}}(\partial f_{k})$ and there exists an integer $M_{k}$ such that

[TABLE]

The first assumption on the domains of the subdifferentials of the functions $(f_{k})_{k\in[\![1,K]\!]}$ is however not satisfied in (69). In this case, the direct simpler form of the algorithm in Combettes (2003)* can be applied since the parameter $\lambda$ is fixed.*

6 Application to robust binary classification

6.1 Context

In this section, we illustrate the performance of our approach on different scenarios in the context of binary classification. To this aim, we consider the ionosphere and colon-cancer datasets 222https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html.. The respective numbers of observations $N$ and of features $d$ are summarized in Table 1. Unless specified, we will consider the original datasets without pre-processing, using a training set with 60% of the original database and a testing set gathering the remaining entries. The splitting between training and testing samples is performed using function train_test_split of Scikit-learn 333https://scikit-learn.org. We propose to compare the classical formulation in Equation (2) with the formulation in Problem 9 (resp. Problem 12) that uses ambiguity sets defined through the Kullback-Leibler divergence (resp. Wasserstein distance). We make use of the logistic regression loss in (4) (Briceno-Arias et al. (2017)* for recent developments). The constrained minimization problems are solved running the proposed Algorithm 1 over a sufficient number of iterations so as to reach the stability criterion $\|p^{(\ell+1)}-p^{(\ell)}\|\leq 10^{-5}$. All the tests are performed by using Julia programming language, on a computer with processors Intel® Core™ i7-3610QM CPU @ 2.30GHz × 8 and 16Gb of RAM.*

6.2 Results in standard conditions

6.2.1 Ionosphere dataset

We display the evolution of the difference between the current cost function and its final value (computed after a very high number of iterations), with respect to the iteration number (see Figure 1) and CPU time (see Figure 2). In the case of the Kullback-Leibler divergence, we choose $\mathbb{K}_{\ell}=[1,K]$ while for Wasserstein distance, we set the cardinality of $\mathbb{K}_{\ell}$ equal to $1500$ ($K=44100$* in this case). In this example, we observe that the convergence speed is slightly increased for large values of $\varepsilon$. Regarding the comparison between the two ambiguity sets, it can be observed on Figure 2 that the method is faster in the case of the Kullback-Leibler divergence since the number of constraints grows linearly as a function of the number of observations, whereas the growth is quadratic in the case of the Wasserstein distance.*

Figure 3 shows the value of the area under the ROC curve (AUC metric) Bradley (1997)* as a function of $\epsilon$, computed using the testing set. There is a clear compromise in the choice of the value of $\epsilon$ for maximizing the AUC, and the best performance are obtained for an intermediate non-zero value of this parameter. This clearly illustrates the benefit of the proposed formulation. Note that such results are consistent with the conclusions in Shafieezadeh-Abadeh, Esfahani, and Kuhn (2015)** and Gotoh, Kim, and Lim (2018)**. On this example, the Wassertein ambiguity set provides better results than the Kullback-Leibler divergence but it should be reminded that it comes at the expense of a higher computational cost.*

6.2.2 Colon-cancer dataset

We now present in Table 2 the evolution of the AUC for tests performed on the colon-cancer dataset. This dataset only contains 64 observations. For such small dataset, the formulation in Problem 12 becomes very cheap in terms of computational cost. As can be noticed in Table 2, taking a nonzero value for $\epsilon$ leads to an increase of about 7% in terms AUC, which is significant in such challenging context. These results allow to assess the robustness property of the proposed formulation.

6.3 Results on an altered database

Let us come back to the processing of ionosphere dataset. In order to better illustrate the interest of the proposed formulation, we propose to modify the training set so that the proportion of labels $(-1)$ and $(+1)$ is altered and unbalanced. Such situation could typically arise during a transient regime, such as the beginning of an epidemic, or in the case of an incomplete dataset. After dividing the dataset between a training set and a testing set, using the same 60% ratio as in our previous tests, we will drop randomly a certain number of observations associated with the label $(-1)$ so that the proportion of this label becomes ten times lower than its original proportions. Figure 4 displays ROC curves and Table 3 evaluates AUC metric, for various values of $\epsilon$. Our formulation clearly outperforms the classical logistic regression classifier (retrieved when $\epsilon=0)$. Noticeably, the later presents the same area under the curve as a random classifier, and thus exhibits a similar behavior to such a classifier.

6.4 Variance reduction study

In a nutshell, the robust framework based on the Wasserstein distance provides a better expected reward but at the expense of a higher computational cost. The risk measure based on the Kullback-Leibler divergence is easily tractable, provides a reduction of the variance in out-of-samples results, but a smaller increase in terms of expected reward (see  Gotoh, Kim, and Lim (2018)* for a more detailed theoretical analysis). In practice, as discussed in  Shafieezadeh-Abadeh, Esfahani, and Kuhn (2015)** and in Gotoh, Kim, and Lim (2018)**, “a little of robustness” typically improves a bit the expected reward (around 1%), however results in a larger reduction in terms of variance. We propose to reproduce such an analysis by means of two experiments using ionosphere dataset. We first consider the case when a small training set is used where only 10% of the data are available. Then we focus on the case when 60% of the data are used as training set. In both cases, 1000 random realizations are run, when we solve the classical formulation using logistic regression (LR) loss in Equation (2), the formulation in Problem 9 that uses ambiguity sets defined through the Kullback-Leibler divergence, and the formulation in Problem 12 that uses ambiguity sets defined through the Wasserstein distance. We then compute the value of the AUC metric on the associated testing set and display results as histograms. When we use 10% of the data for training (Figure 5), we see the benefits of our robust solution with respect to the standard LR classifier. When more data are collected, the probability distribution becomes more accurate and our robust models tend to produce the same outputs as when using classical logistic regression (Figure 6).*

7 Conclusion

We have highlighted that risk measures offer versatile tools for addressing machine learning problems in a robust manner. By assuming that the loss function is convex, the related optimization problem has been recast as a convex one. We have shown that various classes of risk measures, e.g. those based on $\varphi$-divergences or on the Wasserstein distance, lead to a common convex formulation. In addition, an efficient convex optimization algorithm has been proposed to cope with the non trivial constrained problem resulting from this formulation. We have conducted numerical experiments in which various ambiguity sets are tackled thanks to the same algorithm. We have also illustrated that the considered robust models can outperform classical ones in challenging contexts when the size of the training set is limited, or when the distribution of labels in the training set is not representative of the reality.

Acknowledgments

The second author wants to thank Université Paris-Est and Labex Bézout for the financial support particularly for the funding of his PhD program. The work of J.-C. Pesquet was supported by Institut Universitaire de France.