Confidence regions and minimax rates in outlier-robust estimation on the probability simplex

TL;DR

This paper studies robust estimation of a probability distribution's mean on the simplex under adversarial contamination, establishing minimax rates for various distances and providing confidence regions that adapt to sparsity.

Contribution

It derives minimax rates for outlier-robust mean estimation on the probability simplex across multiple distances and constructs adaptive confidence regions.

Findings

Minimax rates differ for total-variation, Hellinger, and L2 distances.

Sample average attains all these minimax rates.

Confidence regions shrink at the minimax rate and adapt to sparsity.

Abstract

We consider the problem of estimating the mean of a distribution supported by the $k$-dimensional probability simplex in the setting where an $\varepsilon$ fraction of observations are subject to adversarial corruption. A simple particular example is the problem of estimating the distribution of a discrete random variable. Assuming that the discrete variable takes $k$ values, the unknown parameter $\boldsymbol \theta$ is a $k$-dimensional vector belonging to the probability simplex. We first describe various settings of contamination and discuss the relation between these settings. We then establish minimax rates when the quality of estimation is measured by the total-variation distance, the Hellinger distance, or the $\mathbb L^2$-distance between two probability measures. We also provide confidence regions for the unknown mean that shrink at the minimax rate. Our analysis reveals that the minimax rates associated to these three distances are all different, but they are all attained by the sample average. Furthermore, we show that the latter is adaptive to the possible sparsity of the unknown vector. Some numerical experiments illustrating our theoretical findings are reported.

Equations

$${\mathbf{P}}_i=(1-\epsilon )\mathbf{P}+\epsilon \mathbf{Q},\forall i\in \{1,\dots ,n\}.$$

$$d_{\text{TV}}(\hat{\mathbf{\theta }},\mathbf{\theta }):=\text{nicefrac}12\Vert \hat{\mathbf{\theta }}-\mathbf{\theta }{\Vert }_1,d_{\text{H}}(\hat{\mathbf{\theta }},\mathbf{\theta }):=\Vert \hat{\mathbf{\theta }}{}^{1/2}-{\mathbf{\theta }}^{1/2}{\Vert }_2,d_{{\mathbb{L}}^2}(\hat{\mathbf{\theta }},\mathbf{\theta }):=\Vert \hat{\mathbf{\theta }}-\mathbf{\theta }{\Vert }_2.$$

$${\mathfrak{R}}_{\square }(n,k,s,\epsilon ):=\underset{{\bar{\mathbf{\theta }}}_n}{\inf }\underset{\mathbf{P},\mathbf{Q}}{\sup }\mathbf{E}[d_{\square }({\bar{\mathbf{\theta }}}_n,\mathbf{\theta })],$$

$${\bar{\mathbf{X}}}_n:=\frac{1}{n}\sum _{i=1}^n{\mathbf{X}}_i$$

$${\mathfrak{R}}_{{\mathbb{L}}^2}(n,k,s,0)\asymp (1/n{)}^{1/2}\text{and}{\mathfrak{R}}_{\square }(n,k,s,0)\asymp (s/n{)}^{1/2}\text{for }\square \in \{\text{TV},\text{H}\}.$$

$${\mathfrak{R}}_{\text{TV}}(n,k,s,\epsilon )$$

$${\mathfrak{R}}_{\text{H}}(n,k,s,\epsilon )$$

$${\mathfrak{R}}_{{\mathbb{L}}^2}(n,k,s,\epsilon )$$

$$\{{\mathbf{X}}_i:i\notin O\}\stackrel{\text{iid}}{\sim }{\mathbf{P}}_{{\mathbf{\theta }}^{\ast }},\{{\mathbf{X}}_i:i\in O\}\stackrel{\text{iid}}{\sim }\mathbf{Q},\{{\mathbf{X}}_i:i\in O\}\perp \perp \{{\mathbf{X}}_i:i\notin O\},$$

$$R_d^{\square }(n,\epsilon ,\Theta ,\hat{\mathbf{\theta }}):=\underset{P_n\in {\mathcal{M}}_n^{\square }(\epsilon ,\Theta )}{\sup }\mathbf{E}[d(\hat{\mathbf{\theta }},{\mathbf{\theta }}^{\ast })].$$

$$R_d^{\text{HC}}(n,\epsilon ,{\mathbf{\theta }}^{\ast },{\hat{\mathbf{\theta }}}_n)\le R_{d,n}^{\text{HDC}}(2\epsilon ,{\mathbf{\theta }}^{\ast },{\hat{\mathbf{\theta }}}_n)+e^{-n\epsilon /3}{R}_{d,n}^{\text{HDC}}(1,{\mathbf{\theta }}^{\ast },{\hat{\mathbf{\theta }}}_n),$$

$$\displaystyle\sup_{\boldsymbol{P}_{n}\in\mathcal{M}_{n}^{\text{HC}}(\varepsilon,\boldsymbol{\theta}^{*})}r\mathbf{P}\big{(}d(\widehat{\boldsymbol{\theta}}_{n},\boldsymbol{\theta}^{*})>r\big{)}\leq R_{d,n}^{\text{HDC}}(2\varepsilon,\boldsymbol{\theta}^{*},\widehat{\boldsymbol{\theta}}_{n})+re^{-n\varepsilon/3}.$$

$$\underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\text{HC}}(n,\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n)\le \underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\text{HDC}}(n,2\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n)+e^{-n\epsilon /3}{\mathcal{D}}_{\Theta }.$$

$$\underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\text{HDC}}(n,\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n)\le \underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\text{OC}}(n,\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n).$$

$$\underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\text{PC}}(n,\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n)\le \underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\mathrm{OC}}(n,\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n)\le \underset{{\hat{\mathbf{\theta }}}_n}{\inf }{R}_d^{\text{AC}}(n,\epsilon ,\Theta ,{\hat{\mathbf{\theta }}}_n),$$

$$R_{\text{TV}}^{\text{AC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{X}}}_n)$$

$$R_{\text{H}}^{\text{AC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{X}}}_n)$$

$$R_{{\mathbb{L}}^2}^{\text{AC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{X}}}_n)$$

$$o_{\text{TV}}^{\ast }(n,s)\asymp (ns{)}^{1/2},o_{\text{H}}^{\ast }(n,s)\asymp s,o_{{\mathbb{L}}^2}^{\ast }(n,s)\asymp n^{1/2}.$$

$$W_q^q(\mathbf{\theta },{\mathbf{\theta }}^{\prime })$$

$$=\underset{\Gamma }{\inf }{2}^{q/2}\mathbf{P}(\mathbf{X}\ne {\mathbf{X}}^{\prime })=2^{q/2}\Vert \mathbf{\theta }-{\mathbf{\theta }}^{\prime }{\Vert }_{\text{TV}},$$

$$R_{W_q}^{\text{AC}}(n,\epsilon ,{\Delta }_s^{k-1})$$

$$\underset{{\bar{\mathbf{\theta }}}_n}{\inf }{R}_{\text{TV}}^{\text{HC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{\theta }}}_n)$$

$$\underset{{\bar{\mathbf{\theta }}}_n}{\inf }{R}_{\text{H}}^{\text{HC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{\theta }}}_n)$$

$$\underset{{\bar{\mathbf{\theta }}}_n}{\inf }{R}_{{\mathbb{L}}^2}^{\text{HC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{\theta }}}_n)$$

$$R_{\text{TV}}^{\text{HDC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{\theta }}}_n)$$

$$\stackrel{(1)}{=}{R}_{\text{TV}}^{\text{HC}}(n,0,{\Delta }_s^{k-1},{\bar{\mathbf{\theta }}}_n)\stackrel{(2)}{\ge }c(s/n{)}^{1/2},$$

$$R_{\text{TV}}^{\text{HDC}}(n,\epsilon ,{\Delta }_s^{k-1},{\bar{\mathbf{\theta }}}_n)$$

$$\displaystyle\geq(c/4)\big{\{}(s/n)^{1/2}+\varepsilon\big{\}}.$$

$$\displaystyle R_{\text{TV}}^{\text{HDC}}(n,\varepsilon,\Delta_{s}^{k-1},\bar{\boldsymbol{\theta}}_{n})\geq(c/4)\big{\{}(s/n)^{1/2}+\varepsilon\big{\}}$$

$$\displaystyle\inf_{\bar{\boldsymbol{\theta}_{n}}}\sup_{\mathcal{M}_{n}^{\rm HC}(\varepsilon,\Delta)}\boldsymbol{P}\big{(}d(\bar{\boldsymbol{\theta}}_{n},{\boldsymbol{\theta}}^{*})\geq w_{d}(\varepsilon,\Delta)\big{)}\geq c_{1},$$

$$\displaystyle d_{\mathbb{L}^{2}}(\bar{\boldsymbol{X}}_{n},\boldsymbol{\theta})\leq(1/n)^{1/2}+\sqrt{2}\,\varepsilon+\big{(}\log(1/\delta)/n\big{)}^{1/2},$$

$$\displaystyle d_{\textup{TV}}(\bar{\boldsymbol{X}}_{n},\boldsymbol{\theta})\leq(s/n)^{1/2}+2\,\varepsilon+\big{(}2\log(1/\delta)/n\big{)}^{1/2},$$

$$\displaystyle d_{\textup{H}}(\bar{\boldsymbol{X}}_{n},\boldsymbol{\theta})\leq 3.2\big{(}(s/n)\log(2s/\delta)\big{)}^{1/2}+(2\varepsilon)^{1/2}+\big{(}(1/n)\log(2/\delta)\big{)}^{1/2},$$

$$\underset{{\mathcal{M}}_n^{\text{HC}}(\epsilon ,{\mathbf{\theta }}^{\ast })}{\sup }\mathbf{E}[d({\hat{\mathbf{\theta }}}_n,{\mathbf{\theta }}^{\ast })\text{mathds}1(\hat{O}=O)]$$

$$\le \underset{{\mathcal{M}}_n^{\text{HDC}}(o/n,{\mathbf{\theta }}^{\ast })}{\sup }\mathbf{E}[d({\hat{\mathbf{\theta }}}_n,{\mathbf{\theta }}^{\ast })]\mathbf{P}(\hat{O}=O)$$

$$\stackrel{(1)}{\le }\underset{{\mathcal{M}}_n^{\text{HDC}}(1,{\mathbf{\theta }}^{\ast })}{\sup }\mathbf{E}[d({\hat{\mathbf{\theta }}}_n,{\mathbf{\theta }}^{\ast })]\mathbf{P}(\hat{O}=O).$$

Full text

62F35,

62H12,

sample,

LaTeX 2ε,

1. 1 Introduction
2. 2 Various models of contamination
3. 3 Prior work
4. 4 Minimax rates on the “sparse” simplex and confidence regions
5. 5 Illustration on a numerical example
6. 6 Summary and conclusion
7. A Proofs of propositions
8. B Upper bounds on the minimax risk over the sparse simplex
9. C Lower bounds on the minimax risk over the sparse simplex
10. D Proofs of bounds with high probability

1 Introduction

Assume $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ are $n$ independent random variables taking their values in the $k$-dimensional probability simplex $\Delta^{k-1}=\{\boldsymbol{v}\in\mathbb{R}_{+}^{k}:v_{1}+\ldots+v_{k}=1\}$. Our goal is to estimate the unknown vector $\boldsymbol{\theta}=\mathbf{E}[\boldsymbol{X}_{i}]$ in the case where the observations are contaminated by outliers. In this introduction, to convey the main messages, we limit ourselves to the Huber contamination model, although our results apply to the more general adversarial contamination. Huber’s contamination model assumes that there are two probability measures $\boldsymbol{P}$, $\boldsymbol{Q}$ on $\Delta^{k-1}$ and a real $\varepsilon\in[0,1/2)$ such that $\boldsymbol{X}_{i}$ is drawn from

[TABLE]

This amounts to assuming that $(1-\varepsilon)$-fraction of observations, called inliers, are drawn from a reference distribution $\boldsymbol{P}$, whereas $\varepsilon$-fraction of observations are outliers and are drawn from another distribution $\boldsymbol{Q}$. In general, all the three parameters $\boldsymbol{P}$, $\boldsymbol{Q}$ and $\varepsilon$ are unknown. The parameter of interest is some functional (such as the mean, the standard deviation, etc.) of the reference distribution $\boldsymbol{P}$, whereas $\boldsymbol{Q}$ and $\varepsilon$ play the role of nuisance parameters.

When the unknown parameter lives on the probability simplex, there are many appealing ways of defining the risk. We focus on the following three metrics: total-variation, Hellinger and $\mathbb{L}^{2}$ distances111We write $\|\boldsymbol{u}\|_{q}=(\sum_{j=1}^{k}|u_{j}|^{q})^{1/q}$ and $\boldsymbol{u}^{q}=(u_{1}^{q},\ldots,u_{k}^{q})$ for any $\boldsymbol{u}\in\mathbb{R}_{+}^{k}$ and $q>0$.

[TABLE]

The Hellinger distance above is well defined when the estimator $\widehat{\boldsymbol{\theta}}$ is non-negative, which will be the case throughout this work. We will further assume that the dimension $k$ may be large, but the vector $\boldsymbol{\theta}$ is $s$-sparse, for some $s\leq k$, i.e. $\#\{j:\theta_{j}\neq 0\}\leq s$. Our main interest is in constructing confidence regions and evaluating the minimax risk

[TABLE]

where the inf is over all estimators $\bar{\boldsymbol{\theta}}_{n}$ built upon the observations $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}\overset{\text{iid}}{\sim}(1-\varepsilon)\boldsymbol{P}+\varepsilon\boldsymbol{Q}$ and the sup is over all distributions $\boldsymbol{P}$, $\boldsymbol{Q}$ on the probability simplex such that the mean $\boldsymbol{\theta}$ of ${\boldsymbol{P}}$ is $s$-sparse. The subscript $\square$ of $\mathfrak{R}$ above refers to the distance used in the risk, so that $\square$ is TV, H, or $\mathbb{L}^{2}$.

The problem described above arises in many practical situations. One example is an election poll: each participant expresses his intention to vote for one of $k$ candidates. Thus, each $\theta_{j}$ is the true proportion of electors of candidate $j$. The results of the poll contain outliers, since some participants of the poll prefer to hide their true opinion. Another example, still related to elections, is the problem of counting votes across all constituencies. Each constituency communicates a vector of proportions to a central office, which is in charge of computing the overall proportions. However, in some constituencies (hopefully a small fraction only) the results are rigged. Therefore, the set of observed vectors contains some outliers.

We intend to provide non-asymptotic upper and lower bounds on the minimax risk that match up to numerical constants. In addition, we will provide confidence regions of the form $B_{\square}(\widehat{\boldsymbol{\theta}}_{n},r_{n,\varepsilon,\delta})=\{\boldsymbol{\theta}:d_{\square}(\widehat{\boldsymbol{\theta}}_{n},\boldsymbol{\theta})\leq r_{n,\varepsilon,\delta}\}$ containing the true parameter with probability at least $1-\delta$ and such that the radius $r_{n,\varepsilon,\delta}$ goes to zero at the same rate as the corresponding minimax risk.

When there is no outlier, i.e., $\varepsilon=0$, it is well known that the sample mean

[TABLE]

is minimax-rate-optimal and the rates corresponding to various distances are

[TABLE]

This raises several questions in the setting where data contains outliers. In particular, the following three questions will be answered in this work:

Q1.

How the risks $\mathfrak{R}_{\square}$ depend on $\varepsilon$? What is the largest proportion of outliers for which the minimax rate is the same as in the outlier-free case ?

Q2.

Does the sample mean remain optimal in the contaminated setting?

Q3.

What happens if the unknown parameter $\boldsymbol{\theta}$ is $s$-sparse ?

The most important step for answering these questions is to show that

[TABLE]

It is surprising to see that all the three rates are different leading to important discrepancies in the answers to the second part of question Q1 for different distances. Indeed, it turns out that the minimax rate is not deteriorated if the proportion of the outliers is smaller than $(s/n)^{1/2}$ for the TV-distance, $s/n$ for the Hellinger distance and $(1/n)^{1/2}$ for the $\mathbb{L}^{2}$ distance. Furthermore, we prove that the sample mean is minimax rate optimal. Thus, even when the proportion of outliers $\varepsilon$ and the sparsity $s$ are known, it is not possible to improve upon the sample mean. In addition, we show that all these claims hold true for the adversarial contamination and we provide corresponding confidence regions.

The rest of the paper is organized as follows. Section 2 introduces different possible ways of modeling data sets contaminated by outliers. Pointers to relevant prior work are given in Section 3. Main theoretical results and their numerical illustration are reported in Section 4 and Section 5, respectively. Section 6 contains a brief summary of the obtained results and their consequences, whereas the proofs are postponed to the appendix.

2 Various models of contamination

Different mathematical frameworks have been used in the literature to model the outliers. We present here five of them, from the most restrictive one to the most general, and describe their relationship. We present these frameworks in the general setting when the goal is to estimate the parameter $\boldsymbol{\theta}^{*}$ of a reference distribution $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$ when $\varepsilon$ proportion of the observations are outliers.

2.1 Huber’s contamination

The most popular framework for studying robust estimation methods is perhaps the one of Huber’s contamination. In this framework, there is a distribution $\boldsymbol{Q}$ defined on the same space as the reference distribution $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$ such that all the observations $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ are independent and drawn from the mixture distribution $\boldsymbol{P}_{\varepsilon,\boldsymbol{\theta}^{*},\boldsymbol{Q}}:=(1-\varepsilon)\boldsymbol{P}_{\boldsymbol{\theta}^{*}}+\varepsilon\boldsymbol{Q}$.

This corresponds to the following mechanism: one decides with probabilities $(1-\varepsilon,\varepsilon)$ whether a given observation is an inlier or an outlier. If the decision is made in favor of being inlier, the observation is drawn from $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$, otherwise it is drawn from $\boldsymbol{Q}$. More formally, if we denote by $\widehat{O}$ the random set of outliers, then conditionally to $\widehat{O}=O$,

[TABLE]

for every $O\subset\{1,\ldots,n\}$. Furthermore, for every subset $O$ of the observations, we have $\boldsymbol{P}(\widehat{O}=O)=(1-\varepsilon)^{n-|O|}\varepsilon^{|O|}$. We denote by222The superscript HC refers to the Huber’s contamination $\mathcal{M}_{n}^{\text{HC}}(\varepsilon,\boldsymbol{\theta}^{*})$ the set of joint probability distributions ${\boldsymbol{P}}\!_{n}$ of the random variables $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ satisfying the foregoing condition.

2.2 Huber’s deterministic contamination

The set of outliers as well as the number of outliers in Huber’s model of contamination are random. This makes it difficult to compare this model to the others that will be described later in this section. To cope with this, we define here another model, termed Huber’s deterministic contamination. As its name indicates, this new model has the advantage of containing a deterministic number of outliers, in the same time being equivalent to Huber’s contamination in a sense that will be made precise below.

We say that the distribution ${\boldsymbol{P}}\!_{n}$ of $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ belongs to the Huber’s deterministic contamination model denoted by $\mathcal{M}_{n}^{\text{HDC}}(\varepsilon,\boldsymbol{\theta}^{*})$, if there are a set $O\subset\{1,\ldots,n\}$ of cardinality at most $n\varepsilon$ and a distribution $\boldsymbol{Q}$ such that (9) is true. The apparent similarity of models $\mathcal{M}_{n}^{\text{HC}}(\varepsilon,\boldsymbol{\theta}^{*})$ and $\mathcal{M}_{n}^{\text{HDC}}(\varepsilon,\boldsymbol{\theta}^{*})$ can also be formalized mathematically in terms of the orders of magnitude of minimax risks. To ease notation, we let $R_{d}^{\square}(n,\varepsilon,\Theta,\widehat{\boldsymbol{\theta}})$ to be the worst-case risk of an estimator $\widehat{\boldsymbol{\theta}}$, where $\square$ is either HC or HDC. More precisely, for $\mathcal{M}_{n}^{\square}(\varepsilon,\Theta):=\cup_{\boldsymbol{\theta}\in\Theta}\mathcal{M}_{n}^{\square}(\varepsilon,\boldsymbol{\theta})$, we set333The subscript $d$ refers to the distance $d$ used in the definition of the risk.

[TABLE]

This definition assumes that the parameter space $\Theta$ is endowed with a pseudo-metric $d:\Theta\times\Theta\to\mathbb{R}_{+}$. When $\Theta=\{\boldsymbol{\theta}^{*}\}$ is a singleton, we write $R_{d,n}^{\square}(\varepsilon,\boldsymbol{\theta}^{*},\widehat{\boldsymbol{\theta}})$ instead of $R_{d}^{\square}(n,\varepsilon,\{\boldsymbol{\theta}^{*}\},\widehat{\boldsymbol{\theta}})$.

Proposition 1**.**

Let $\widehat{\boldsymbol{\theta}}_{n}$ be an arbitrary estimator of $\boldsymbol{\theta}^{*}$. For any $\varepsilon\in(0,1/2)$,

[TABLE]

Proof in the appendix, page A.1

Denote by $\mathcal{D}_{\Theta}$ the diameter of $\Theta$, $\mathcal{D}_{\Theta}:=\max_{\boldsymbol{\theta},\boldsymbol{\theta}^{\prime}}d(\boldsymbol{\theta},\boldsymbol{\theta}^{\prime})$. Proposition LABEL:prop@2 implies that

[TABLE]

When $\Theta$ is bounded, the last term is typically of smaller order than the minimax risk over $\mathcal{M}^{\text{HDC}}_{n}(2\varepsilon,\Theta)$. Therefore, the minimax rate of estimation in Huber’s model is not slower than the minimax rate of estimation in Huber’s deterministic contamination model. This entails that a lower bound on the minimax risk established in HC-model furnishes a lower bound in HDC-model.

2.3 Oblivious contamination

A third model of contamination that can be of interest is the oblivious contamination. In this model, it is assumed that the set $O$ of cardinality $o$ and the joint distribution $\boldsymbol{Q}_{O}$ of outliers are determined in advance, possibly based on the knowledge of the reference distribution $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$. Then, the outliers $\{\boldsymbol{X}_{i}:i\in O\}$ are drawn randomly from $\boldsymbol{Q}_{O}$ independently of the inliers $\{\boldsymbol{X}_{i}:i\in O^{c}\}$. The set of all the joint distributions $\boldsymbol{P}_{n}$ of random variables $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ generated by such a mechanism will be denoted by $\mathcal{M}_{n}^{\text{OC}}(\varepsilon,\boldsymbol{\theta}^{*})$. The model of oblivious contamination is strictly more general than that of Huber’s deterministic contamination, since it does not assume that the outliers are iid. Therefore, the minimax risk over $\mathcal{M}_{n}^{\text{OC}}(\varepsilon,\Theta)$ is larger than the minimax risk over $\mathcal{M}_{n}^{\text{HDC}}(\varepsilon,\Theta)$:

[TABLE]

The last inequality holds true for any set $\Theta$, any contamination level $\varepsilon\in(0,1)$ and any sample size.

2.4 Parameter contamination

In the three models considered above, the contamination acts on the observations. One can also consider the case where the parameters of the distributions of some observations are contaminated. More precisely, for some set $O\subset\{1,\ldots,n\}$ selected in advance (but unobserved), the outliers $\{\boldsymbol{X}_{i}:i\in O\}$ are independent and independent of the inliers $\{\boldsymbol{X}_{i}:i\in O^{c}\}$. Furthermore, each outlier $\boldsymbol{X}_{i}$ is drawn from a distribution $\boldsymbol{Q}_{i}=\boldsymbol{P}_{\boldsymbol{\theta}_{i}}$ belonging to the same family as the reference distribution, but corresponding to a contaminated parameter $\boldsymbol{\theta}_{i}\not=\boldsymbol{\theta}^{*}$. Thus, the joint distribution of the observations can be written as $(\bigotimes_{i\in O^{c}}\boldsymbol{P}_{\boldsymbol{\theta}^{*}})\otimes(\bigotimes_{i\in O}\boldsymbol{P}_{\boldsymbol{\theta}_{i}})$. The set of all such distributions $\boldsymbol{P}_{n}$ will be denoted by $\mathcal{M}_{n}^{\text{PC}}(\varepsilon,\boldsymbol{\theta}^{*})$, where PC refers to “parameter contamination”.

2.5 Adversarial contamination

The last model of contamination we describe in this work, the adversarial contamination, is the most general one. It corresponds to the following two-stage data generation mechanism. In a first stage, iid random variables $\boldsymbol{Y}\!_{1},\ldots,\boldsymbol{Y}\!_{n}$ are generated from a reference distribution $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$. In a second stage, an adversary having access to $\boldsymbol{Y}\!_{1},\ldots,\boldsymbol{Y}\!_{n}$ chooses a (random) set $\widehat{O}$ of (deterministic) cardinality $s$ and arbitrarily modifies data points $\{\boldsymbol{Y}\!_{i}:i\in\widehat{O}\}$. The resulting sample, $\{\boldsymbol{X}_{i}:i=1,\ldots,n\}$, is revealed to the Statistician. In this model, we have $\boldsymbol{X}_{i}=\boldsymbol{Y}\!_{i}$ for $i\not\in\widehat{O}$. However, since $\widehat{O}$ is random and potentially dependent of $\boldsymbol{Y}\!_{1:n}$, it is not true that conditionally to $\widehat{O}=O$, $\{\boldsymbol{X}_{i}:i\in O^{c}\}$ are iid drawn from $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$ (for any deterministic set $O$ of cardinality $o$).

We denote by $\mathcal{M}_{n}^{\text{AC}}(\varepsilon,\boldsymbol{\theta}^{*})$ the set of all the joint distributions $\boldsymbol{P}_{n}$ of all the sequences $\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n}$ generated by the aforementioned two-stage mechanism. This set $\mathcal{M}_{n}^{\text{AC}}(\varepsilon,\boldsymbol{\theta}^{*})$ is larger than all the four sets of contamination introduced in this section. Therefore, the following inequalities hold:

[TABLE]

for any $n$, $\varepsilon$, $\Theta$ and any distance $d$.

2.6 Minimax risk “in expectation” versus “in deviation”

Most prior work on robust estimation focused on establishing upper bounds on the minimax risk in deviation444We call a risk bound in deviation any bound on the distance $d(\widehat{\boldsymbol{\theta}},\boldsymbol{\theta}^{*})$ that holds true with a probability close to one, for any parameter value $\boldsymbol{\theta}^{*}\in\Theta$., as opposed to the minimax risk in expectation defined by (3). One of the reasons for dealing with the deviation is that it makes the minimax risk meaningful for models555This is the case, for instance, of the Gaussian model with Huber’s contamination. having random number of outliers and unbounded parameter space $\Theta$. The formal justification of this claim is provided by the following result.

Proposition 2**.**

Let $\Theta$ be a parameter space such that $\mathcal{D}_{\Theta}=\sup_{\boldsymbol{\theta},\boldsymbol{\theta}^{\prime}\in\Theta}d(\boldsymbol{\theta},\boldsymbol{\theta}^{\prime})=+\infty$. Then, for every estimator $\widehat{\theta}_{n}$, every $\varepsilon>0$ and $n\in\mathbb{N}$, we have $R_{d}^{\text{HC}}(n,\varepsilon,\Theta,\widehat{\boldsymbol{\theta}}_{n})=+\infty$.

Proof in the appendix, page A.2

This result shows, in particular, that the last term in (13), involving the diameter of $\Theta$ is unavoidable. Such an explosion of the minimax risk occurs because Huber’s model allows the number of outliers to be as large as $n/2$ with a strictly positive probability. One approach to overcome this shortcoming is to use the minimax risk in deviation. Another approach is to limit theoretical developments to the models HDC, PC, OC or AC, in which the number of outliers is deterministic.

3 Prior work

Robust estimation is an area of active research in Statistics since at least five decades (Huber, 1964; Tukey, 1975; Donoho and Huber, 1983; Donoho and Gasko, 1992; Rousseeuw and Hubert, 1999). Until very recently, theoretical guarantees were almost exclusively formulated in terms of the notions of breakdown point, sensitivity curve, influence function, etc. These notions are well suited for accounting for gross outliers, observations that deviate significantly from the data points representative of an important fraction of data set.

More recently, various authors investigated (Nguyen and Tran, 2013; Dalalyan and Chen, 2012; Chen et al., 2013) the behavior of the risk of robust estimators as a function of the rate of contamination $\varepsilon$. A general methodology for parametric models subject to Huber’s contamination was developed in Chen et al. (2018, 2016). This methodology allowed for determining the rate of convergence of the minimax risk as a function of the sample size $n$, dimension $k$ and the rate of contamination $\varepsilon$. An interesting phenomenon was discovered: in the problem of robust estimation of the Gaussian mean, classic robust estimators such as the coordinatewise median or the geometric median do not attain the optimal rate $(k/n)^{1/2}+\varepsilon$. This rate is provably attained by Tukey’s median, the computation of which is costly in a high dimensional setting.

In the model analyzed in this paper, we find the same minimax rate, $(k/n)^{1/2}+\varepsilon$, only when the total-variation distance is considered. A striking difference is that this rate is attained by the sample mean which is efficiently computable in any dimension. This property is to some extent similar to the problem of robust density estimation (Liu and Gao, 2017), in which the standard kernel estimators are minimax optimal in contaminated setting.

Computational intractability of Tukey’s median motivated a large number of studies that aimed at designing computationally tractable methods with nearly optimal statistical guarantees. Many of these works went beyond Huber’s contamination by considering parameter contamination models (Bhatia et al., 2017; Collier and Dalalyan, 2017; Carpentier et al., 2018), oblivious contamination (Feng et al., 2014; Lai et al., 2016) or adversarial contamination (Diakonikolas et al., 2016; Balakrishnan et al., 2017; Diakonikolas et al., 2017, 2018). Interestingly, in the problem of estimating the Gaussian mean, it was proven that the minimax rates under adversarial contamination are within a factor at most logarithmic in $n$ and $k$ of the minimax rates under Huber’s contamination666All these papers consider the risk in deviation, so that the minimax risk under Huber’s contamination is finite.. While each of the aforementioned papers introduced clearly the conditions on the contamination, to our knowledge, none of them described different possible models and the relationship between them.

Another line of growing literature on robust estimation aims at robustifying estimators and prediction methods to heavy tailed distributions, see (Audibert and Catoni, 2011; Minsker, 2015; Donoho and Montanari, 2016; Devroye et al., 2016; Joly et al., 2017; Minsker, 2018; Lugosi and Mendelson, 2019; Lecué and Lerasle, 2017; Chinot et al., 2018). The results of those papers are of a different nature, as compared to the present work, not only in terms of the goals, but also in terms of mathematical and algorithmic tools.

4 Minimax rates on the “sparse” simplex

and confidence regions

We now specialize the general setting of Section 2 to a reference distribution $\boldsymbol{P}$, with expectation $\boldsymbol{\boldsymbol{\theta}}^{*}$, defined on the simplex $\Delta^{k-1}$. Along with this reference model describing the distribution of inliers, we will use different models of contamination. More precisely, we will establish upper bounds on worst-case risks of the sample mean in the most general, adversarial, contamination setting. Then, matching lower bounds will be provided for minimax risks under Huber’s contamination.

4.1 Upper bounds: worst-case risk of the sample mean

We denote by $\Delta_{s}^{k-1}$ the set of all $\boldsymbol{v}\in\Delta^{k-1}$ having at most $s$ non-zero entries.

Theorem 3**.**

For every triple of positive integers $(k,s,n)$ and for every $\varepsilon\in[0,1]$, the sample mean $\bar{\boldsymbol{X}}_{n}:=\frac{1}{n}\sum_{i=1}^{n}\boldsymbol{X}_{i}$ satisfies

[TABLE]

Proof in the appendix, page B.1

An unexpected and curious phenomenon unveiled by this theorem is that all the three rates are different. As a consequence, the answer to the question “what is the largest possible number of outliers, $o_{d}^{*}(n,s)$, that does not impact the minimax rate of estimation of $\boldsymbol{\theta}^{*}$?” crucially depends on the considered distance $d$. Taking into account the relation $\varepsilon=o/n$, we get

[TABLE]

Furthermore, all the claims concerning the total variation distance, in the considered model, yield corresponding claims for the Wasserstein distances $W_{q}$, for every $q\geq 1$. Indeed, one can see an element $\boldsymbol{\theta}\in\Delta^{k-1}$ as the probability distribution of a random vector $\boldsymbol{X}$ taking values in the finite set $\mathcal{A}=\{\boldsymbol{e}_{1},\ldots,\boldsymbol{e}_{k}\}$ of vectors of the canonical basis of $\mathbb{R}^{k}$. Since these vectors satisfy $\|\boldsymbol{e}_{j}-\boldsymbol{e}_{j^{\prime}}\|^{2}_{2}=2\mathds{1}(j\neq j^{\prime})$, we have

[TABLE]

where the inf is over all joint distributions $\Gamma$ on $\mathcal{A}\times\mathcal{A}$ having marginal distributions $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$. This implies that

[TABLE]

In addition, since the $\mathbb{L}_{2}$ norm is an upper bound on the $\mathbb{L}_{\infty}$-norm, we have $R_{\mathbb{L}_{\infty}}^{\text{AC}}(n,\varepsilon,\Delta^{k-1})\leq(1/n)^{1/2}+\sqrt{2}\,\varepsilon$. Thus, we have obtained upper bounds on the risk of the sample mean for all commonly used distances on the space of probability measures.

4.2 Lower bounds on the minimax risk

A natural question, answered in the next theorem, is how tight are the upper bounds obtained in the last theorem. More importantly, one can wonder whether there is an estimator that has a worst-case risk of smaller order than that of the sample mean.

Theorem 4**.**

There are universal constants $c>0$ and $n_{0}$, such that for any integers $k\geq 3$, $s\leq k\wedge n$, $n\geq n_{0}$ and for any $\varepsilon\in[0,1]$, we have

[TABLE]

where $\inf_{\bar{\boldsymbol{\theta}}_{n}}$ stands for the infimum over all measurable functions $\bar{\boldsymbol{\theta}}_{n}$ from $(\Delta^{k-1})^{n}$ to $\Delta^{k-1}$.

Proof in the appendix, page C.1

The main consequence of this theorem is that whatever the contamination model is (among those described in Section 2), the rates obtained for the MLE in 3 are minimax optimal. Indeed, Theorem LABEL:theorem@2 yields this claim for Huber’s contamination. For Huber’s deterministic contamination and and the TV-distance, on the one hand, we have

[TABLE]

where (1) uses the fact that for $\varepsilon=0$ all the sets $\mathcal{M}_{n}^{\square}(\varepsilon,\boldsymbol{\theta}^{*})$ are equal, while (2) follows from the last theorem. On the other hand, in view of Proposition LABEL:prop@1, for $\varepsilon\geq(6/n)\log(8n/c)$ (implying that $2e^{-n\varepsilon/6}\leq(c/4)\varepsilon$),

[TABLE]

Combining these two inequalities, for $n\geq(10+2\log(1/c))^{2}$, we get

[TABLE]

for every $k\geq 1$ and every $\varepsilon\in[0,1]$. The same argument can be used to show that all the inequalities in Theorem LABEL:theorem@2 are valid for Huber’s deterministic contamination model as well. Since the inclusions $\mathcal{M}_{n}^{\text{HDC}}(\varepsilon,\boldsymbol{\theta}^{*})\subset\mathcal{M}_{n}^{\text{OC}}(\varepsilon,\boldsymbol{\theta}^{*})\cap\mathcal{M}_{n}^{\text{PC}}(\varepsilon,\boldsymbol{\theta}^{*})\subset\mathcal{M}_{n}^{\text{AC}}(\varepsilon,\boldsymbol{\theta}^{*})$ hold true, we conclude that the lower bounds obtained for HC remain valid for all the other contamination models and are minimax optimal.

The main tool in the proof of Theorem LABEL:theorem@2 is the following result (Chen et al., 2018, Theorem 5.1). There is a universal constant $c_{1}>0$ such that for every $\varepsilon\in[0,1)$,

[TABLE]

where $w_{d}(\varepsilon,\Delta)$ is the modulus of continuity defined by $w_{d}(\varepsilon,\Delta)=\sup\{d(\boldsymbol{\theta},\boldsymbol{\theta}^{\prime}):d_{\text{TV}}(\boldsymbol{\theta},\boldsymbol{\theta}^{\prime})\leq\varepsilon/(1-\varepsilon)\}$. Choosing $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$ to differ on only to coordinates, one can check that, for any $\varepsilon\leq 1/2$, $w_{\rm TV}(\varepsilon,\Delta_{s}^{k-1})\geq\varepsilon$, $w_{\rm H}(\varepsilon,\Delta_{s}^{k-1})\geq(\varepsilon/2)^{1/2}$ and $w_{\mathbb{L}^{2}}(\varepsilon,\Delta_{s}^{k-1})\geq\sqrt{2}\varepsilon$. Combining with the lower bounds in the non-contaminated setting, this result yields the claims of Theorem LABEL:theorem@2. In addition, (20) combined with the results of this section implies that the rate in (22) is minimax optimal.

4.3 Confidence regions

We established so far bounds for the expected value of estimation error. The aim of this section is to present bounds on estimation error of the sample mean holding with high probability. This also leads to constructing confidence regions for the parameter vector $\boldsymbol{\theta}^{*}$. To this end, the contamination rate $\varepsilon$ and the sparsity $s$ are assumed to be known. It is an interesting open question whether one can construct optimally shrinking confidence regions for unknown $\varepsilon$ and $s$.

Theorem 5**.**

Let $\delta\in(0,1)$ be the tolerance level. If $\boldsymbol{\theta}^{*}\in\Delta_{s}^{k-1}$, then under any contamination model, the regions of $\Delta^{k-1}$ defined by each of the following inequalities

[TABLE]

contain $\boldsymbol{\theta}^{*}$ with probability at least $1-\delta$.

To illustrate the shapes of these confidence regions, we depicted them in Figure 2 for a three dimensional example, projected onto the plane containing the probability simplex. The sample mean in this example is equal to $(1/3,1/2,1/6)$.

5 Illustration on a numerical example

We provide some numerical experiments which illustrate theoretical results of Section 4. The data set is the collection of 38 books written by Alexandre Dumas (1802-1870) and 38 books written by Emile Zola (1840-1902)777The works of both authors are available from https://www.gutenberg.org/. To each author, we assign a parameter vector corresponding to the distribution of the number of words contained in the sentences used in the author’s books. To be more clear, a sentence containing $l$ words is represented by vector $\boldsymbol{e}_{l}$, and if the parameter vector of an author is $(\theta_{1},\dots,\theta_{k})$, it means that a sentence used by the author is of size $l\in\{1,\dots,k\}$ with probability $\theta_{l}$. We carried out synthetic experiments in which the reference parameter to estimate is the probability vector of Dumas, while the distribution of outliers is determined by the probability vector of Zola. Ground truths for these parameters are computed from the aforementioned large corpus of their works. Only the dense case $s=k$ were considered. For various values of $\varepsilon$ and $n$, a contaminated sample was generated by randomly choosing $n$ sentences either from Dumas’ works (with probability $1-\varepsilon$) or from Zola’s works (with probability $\varepsilon$). The sample mean was computed for this corrupted sample, and the error with respect to Dumas’ parameter vector was measured by the three distances TV, $\mathbb{L}^{2}$ and Hellinger. This experiment was repeated $10^{4}$ times for each special setting to obtain information on error’s distribution. Furthermore, by grouping nearby outcomes we created samples of different dimensions for illustrating the behavior of the error as a function of $k$.

The error of $\bar{X}_{n}$ as a function of the sample size $n$, dimension $k$, and contamination rate $\varepsilon$ is plotted in Figures 3 and 4. These plots are conform to the theoretical results. Indeed, the first plot in Figure 3 shows that the errors for the three distances is decreasing w.r.t.  $n$. Furthermore, we see that up to some level of $n$ this decay is of order $n^{-1/2}$. The second plot in Figure 3 confirms that the risk grows linearly in $k$ for the TV and Hellinger distances, while it is constant for the $\mathbb{L}^{2}$ error.

Left panel of Figure 4 suggests that the error grows linearly in terms of contamination rate. This is conform to our results for the TV and $\mathbb{L}^{2}$ errors. But it might seem that there is a disagreement with the result for the Hellinger distance, for which the risk is shown to increase at the rate $\varepsilon^{1/2}$ and not $\varepsilon$. This is explained by the fact that the rate $\varepsilon^{1/2}$ corresponds to the worst-case risk, whereas here, the setting under experiment does not necessarily represent the worst case. When the parameter vectors of the reference and contamination distributions, respectively, are $\boldsymbol{e}_{i}$ and $\boldsymbol{e}_{j}$ with $i\neq j$ (i.e., when these two distributions are at the largest possible distance, which we call an extreme case), the graph of the error as a function of $\varepsilon$ (right panel of Figure 4) is similar to that of square-root function.

6 Summary and conclusion

We have analyzed the problem of robust estimation of the mean of a random vector belonging to the probability simplex. Four measures of accuracy have been considered: total variation, Hellinger, Euclidean and Wasserstein distances. In each case, we have established the minimax rates of the expected error of estimation under the sparsity assumption. In addition, confidence regions shrinking at the minimax rate have been proposed.

An intriguing observation is that the choice of the distance has much stronger impact on the rate than the nature of contamination. Indeed, while the rates for the aforementioned distances are all different, the rate corresponding to one particular distance is not sensitive to the nature of outliers (ranging from Huber’s contamination to the adversarial one). While the rate obtained for the TV-distance coincides with the previously known rate of robustly estimating a Gaussian mean, the rates we have established for the Hellinger and for the Euclidean distances appear to be new. Interestingly, when the error is measured by the Euclidean distance, the quality of estimation does not get deteriorated with increasing dimension.

Appendix A Proofs of propositions

Proof A.1** (Proof of Proposition LABEL:prop@1 on page 1).**

Recall that $\widehat{O}$ is the set of outliers in the Huber model. Let $O$ be any subset of $\{1,\ldots,n\}$. It follows from the definition of Huber’s model that if $\boldsymbol{P}_{n}^{O}$ stands for the conditional distribution of $(\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n})$ given $\widehat{O}=O$, when $(\boldsymbol{X}_{1},\ldots,\boldsymbol{X}_{n})$ is drawn from $\boldsymbol{P}^{n}\in\mathcal{M}_{n}^{\text{HC}}(\varepsilon,\boldsymbol{\theta}^{*})$, then $\boldsymbol{P}_{n}^{O}\in\mathcal{M}_{n}^{\text{HDC}}(|O|/n,\boldsymbol{\theta}^{*})$. Therefore, for every $O$ of cardinality $o\geq 2\varepsilon n$, we have

[TABLE]

Inequality (1) above is a direct consequence of the inclusion $\mathcal{M}_{n}^{\text{HDC}}(o/n,\boldsymbol{\theta}^{*})\subset\mathcal{M}_{n}^{\text{HDC}}(1,\boldsymbol{\theta}^{*})$. Summing the obtained inequality over all sets $O$ of cardinality $\geq 2\varepsilon n$, we get

[TABLE]

It follows from the multiplicative form of Chernoff’s inequality that $\mathbf{P}(|\widehat{O}|\geq 2\varepsilon n)\leq e^{-n\varepsilon/3}$. This leads to the last term in inequality (11).

Using the same argument as for (36), for any $O$ of cardinality $o<2n\varepsilon$, we get

[TABLE]

This completes the proof of (11).

One can use the same arguments along with the Tchebychev inequality to establish (12). Indeed, for every $S$ of cardinality $o\leq 2\varepsilon n$, we have

[TABLE]

Summing the obtained inequality over all sets $O$ of cardinality $o\leq 2\varepsilon n$, we get

[TABLE]

On the other hand, it holds that

[TABLE]

and the claim of the proposition follows.

Proof A.2** (Proof of Proposition LABEL:prop@2 on page 2).**

Let $\boldsymbol{\theta}_{1}$ and $\boldsymbol{\theta}_{2}$ be two points in $\Theta$. We have

[TABLE]

To ease writing, assume that $n$ is an even number. Let $O$ be any fixed set of cardinality $n/2$. It is clear that the set of outliers $\widehat{O}$ satisfies

[TABLE]

Furthermore, if $\boldsymbol{X}_{1:n}$ is drawn from $((1-\varepsilon)\boldsymbol{P}_{\boldsymbol{\theta}_{1}}+\varepsilon\boldsymbol{P}_{\boldsymbol{\theta}_{2}})^{\otimes n}$, then its conditional distribution given $\widehat{O}=O$ is exactly the same as the conditional distribution of $\boldsymbol{X}_{1:n}\sim((1-\varepsilon)\boldsymbol{P}_{\boldsymbol{\theta}_{2}}+\varepsilon\boldsymbol{P}_{\boldsymbol{\theta}_{1}})^{\otimes n}$ given $\widehat{O}=O^{c}$. This implies that

[TABLE]

where in the last step we have used the triangle inequality. The obtained inequality being true for every $\boldsymbol{\theta}_{1},\boldsymbol{\theta}_{2}\in\Theta$, we can take the supremum to get

[TABLE]

This completes the proof.

Appendix B Upper bounds on the minimax risk over the sparse simplex

This section is devoted to the proof of the upper bounds on minimax risks in the discrete model with respect to various distances.

Proof B.1** (Proof of Theorem LABEL:theorem@1 on page 3).**

To ease notation, we set

[TABLE]

In the adversarial model, we have $\boldsymbol{X}_{i}=\boldsymbol{Y}\!_{i}$ if $i\not\in O$ where $\boldsymbol{Y}\!_{1},\dots,\boldsymbol{Y}\!_{n}$ are generated from the reference distribution $\boldsymbol{\theta}^{*}$.

[TABLE]

which gives us

[TABLE]

And for a fixed $\boldsymbol{\theta}^{*}$ it is well known that

[TABLE]

Hence, we obtain $R_{\rm\mathbb{L}^{2}}^{\rm AC}(n,\varepsilon,\Delta^{k-1})\leq(1/n)^{1/2}+\varepsilon$. Similarly,

[TABLE]

This gives

[TABLE]

In addition, for every $\boldsymbol{\theta}^{*}$,

[TABLE]

where in (1) we have used the notation $J=\{j:\theta_{j}^{*}\neq 0\}$ and in (2) we have used the Cauchy-Schwarz inequality. This leads to

[TABLE]

Finally, for the Hellinger distance

[TABLE]

where we have already seen that

[TABLE]

This yields

[TABLE]

Furthermore, for every $\boldsymbol{\theta}^{*}\in\Delta^{k-1}_{s}$,

[TABLE]

Hence, by Jensen’s inequality $\mathbf{E}[d_{\rm H}(\bar{\boldsymbol{Y}}_{n},\boldsymbol{\theta}^{*})]<\sqrt{s/n}$. Therefore, we infer that

[TABLE]

and the last claim of the theorem follows.

Appendix C Lower bounds on the minimax risk over the sparse simplex

This section is devoted to the proof of the lower bounds on minimax risks in the discrete model with respect to various distances. Note that the rates over the high-dimensional “sparse” simplex $\Delta^{k-1}_{s}$ coincide with those for the dense simplex $\Delta^{s-1}$. For this reason, all the lower bounds will be proved for $\Delta^{s-1}$ only (for $s\geq 2$ an even integer). In addition, we will restrict our attention to the distributions $\boldsymbol{P}$, $\boldsymbol{Q}$ over $\Delta^{s-1}$ that are supported by the set $\mathcal{A}$ of the elements of the canonical basis that is $\boldsymbol{P}(\mathcal{A})=\boldsymbol{Q}(\mathcal{A})=1$.

Proof C.1** (Proof of Theorem LABEL:theorem@2 on page 4).**

We denote by $\boldsymbol{e}_{j}$ the vector in $\mathbb{R}^{s}$ having all the coordinates equal to zero except the $j$th coordinate which is equal to one. Setting

[TABLE]

we have

[TABLE]

Therefore, modulus of continuity defined by

[TABLE]

for a distance $d$ and a set $\Delta$, satisfies for any $\varepsilon\leq 1/2$,

[TABLE]

These bounds on the modulus of continuity are the first ingredient we need to be able to apply Theorem 5.1 from (Chen et al., 2018) for getting minimax lower bounds.

The second ingredient is the minimax rate in the non-contaminated case. are well known. For each of the considered distances, this rate is well-known. However, for the sake of completeness, we provide below a proof those lower bounds using Fano’s method. For this, we use the Varshamov-Gilbert lemma (see e.g. Lemma 2.9 in (Tsybakov, 2009)) and Theorem 2.5 in (Tsybakov, 2009). The Varshamov-Gilbert lemma guarantees the existence of a set ${\boldsymbol{\omega}}^{(1)},\dots,{\boldsymbol{\omega}}^{(M)}\in\{0,1\}^{\lfloor s/2\rfloor}$ of cardinality $M\geq 2^{s/16}$ such that

[TABLE]

where $\rho(.,.)$ stands for the Hamming distance. Using these binary vectors $\{\boldsymbol{\omega}_{j}\}$, a parameter $\beta\in[0,\sqrt{n/s}]$ to be specified later and the “baseline” vector $\boldsymbol{\theta}^{(0)}=(1/s,\dots,1/s)$, we define hypotheses $\boldsymbol{\theta}^{(1)}$, $\dots$, $\boldsymbol{\theta}^{(M)}$ by the relations

[TABLE]

Remark that $\boldsymbol{\theta}^{(0)},\dots,\boldsymbol{\theta}^{(M)}$ are all probability vectors of dimension $s$. Denoting the Kullback-Leibler divergence by $d_{\rm KL}(.,.)$, one can check the conditions of Theorem 2.5 in Tsybakov (2009):

[TABLE]

as well as

[TABLE]

for $\beta=\sqrt{\alpha}/4$. Now by applying the aforementioned theorem, we obtain for the non-contaminated setting ($\varepsilon=0$)

[TABLE]

Setting $M=2^{k/16}$ and $\alpha=1/32$, by Markov’s inequality, one concludes

[TABLE]

where $c=1/25600$. Since, $d_{\rm H}(\boldsymbol{\theta},\boldsymbol{\theta}^{\prime})\geq d_{\rm TV}(\boldsymbol{\theta},\boldsymbol{\theta}^{\prime})$ for any distribution $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$

[TABLE]

Finally, we apply (Chen et al., 2018, Theorem 5.1) stating in our case for any distance $d$

[TABLE]

for an universal constant $c$, which completes the proof of Theorem LABEL:theorem@2.

Appendix D Proofs of bounds with high probability

Proof D.1**.**

Suppose $\boldsymbol{X}_{i}=\boldsymbol{Y}\!_{i}$ if $i\not\in O$ where $\boldsymbol{Y}\!_{1},\dots,\boldsymbol{Y}\!_{n}$ are independently generated from the reference distribution $\boldsymbol{P}$ so that $\mathbf{E}[\boldsymbol{Y}\!_{i}]=\boldsymbol{\theta}^{*}$. For any $\boldsymbol{Z}_{1},\dots,\boldsymbol{Z}_{n}$, let $\boldsymbol{\Phi}_{\square}(\boldsymbol{Z}_{1},\dots,\boldsymbol{Z}_{n}):=d_{\square}(\sum_{i=1}^{n}\boldsymbol{Z}_{i}/n,\boldsymbol{\theta}^{*})$, where $\square$ refers here to the distances $\mathbb{L}^{2}$ or TV. Given $\boldsymbol{Y}\!^{\prime}_{1},\dots,\boldsymbol{Y}\!^{\prime}_{n}\in\Delta^{k-1}$ we have for every $i$

[TABLE]

Furthermore, it can easily be shown that the last term is bounded by $\sqrt{2}/n$ and $2/n$ for the distances $\mathbb{L}^{2}$ and TV, respectively. By bounded difference inequality (see for example Theorem 6.2 of Boucheron et al. (2013)) with probability at least $1-\delta$

[TABLE]

Using $\boldsymbol{\Phi}_{\square}(\boldsymbol{X}_{1},\dots,\boldsymbol{X}_{n})\leq\boldsymbol{\Phi}_{\square}(\boldsymbol{Y}\!_{1},\dots,\boldsymbol{Y}\!_{n})+d_{\square}(\sum_{i\in O}\boldsymbol{X}_{i}/n,\sum_{i\in O}\boldsymbol{Y}\!_{i}/n)$, one can conclude the proof of the first two claims of the theorem.

For the Hellinger distance, the computations are more tedious. We have to separate the case of small $\theta^{*}_{j}$. To this end, let $J=\{j:0<\theta^{*}_{j}<(1/n)\log(2s/\delta)\}$ and $J^{\prime}=\{j:\theta^{*}_{j}\geq(1/n)\log(2s/\delta)\}$. We have

[TABLE]

Since the random variables U_{i}:=\Big{(}\sum_{j\in J}\boldsymbol{Y}\!_{i,j}\Big{)} are iid, positive, bounded by $1$, the Bernstein inequality implies that

[TABLE]

holds with probability at least $1-\delta/2$ for $0<\delta<1$. One easily checks that $\sqrt{\textbf{Var}(U_{1})}\leq\sum_{j\in J}\sqrt{\textbf{Var}(\boldsymbol{Y}\!_{1,j})}\leq s\sqrt{\log(2s/\delta)/n}$. Therefore, with probability at least $1-\delta/2$, we have

[TABLE]

This yields

[TABLE]

with probability at least $1-\delta/2$.

On the other hand, we have

[TABLE]

The Bernstein inequality and the union bound imply that, with probability at least $1-\delta/2$,

[TABLE]

Combining (105) and (108), we obtain

[TABLE]

with probability at least $1-\delta/2$. Finally, inequalities (103) and (110) together lead to

[TABLE]

which is true with probability at least $1-\delta$. Using the triangle inequality and the fact that the Hellinger distance is smaller than the square root of the TV-distance, we get

[TABLE]

with probability at least $1-\delta$. This completes the proof of the theorem.