On the optimality of the Monte-Carlo estimator

TL;DR

This paper proves that for Monte-Carlo estimators on atomless probability spaces, choosing independent random points minimizes the worst-case mean squared error, establishing an optimality condition.

Contribution

It provides a theoretical proof that independence in sampling yields the minimal worst-case mean squared error for Monte-Carlo estimators.

Findings

Independence minimizes worst-case mean squared error.

Optimality holds on atomless probability spaces.

The result guides best practices in Monte-Carlo sampling.

Abstract

We prove that on an atomless probability space, the worst-case mean squared error of the Monte-Carlo estimator is minimal if the random points are chosen independently.

Equations

$$\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^{n}f(Z_i)-{\int }_{X}f\mathrm{d}\mu \right)=\frac{1}{n}{\Vert f-{\int }_{X}f\mathrm{d}\mu \Vert }_{L^2(X,\mu )}^2$$

$$\underset{\begin{array}{c}f\in L^2(X,\mu )\\ \Vert f{\Vert }_2=1\end{array}}{\sup }\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^{n}f(Z_i)-{\int }_{X}f\mathrm{d}\mu \right)=\frac{1}{n}.$$

$$\underset{\begin{array}{c}f\in L^2(X,\mu )\\ \Vert f{\Vert }_2=1\end{array}}{\sup }\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^{n}f(Z_i)-{\int }_{X}f\mathrm{d}\mu \right)\ge \frac{1}{n}\left(1-\frac{n-1}{N-1}\right).$$

$$\underset{\begin{array}{c}f\in L^2(X,\mu )\\ \Vert f{\Vert }_2=1\end{array}}{\sup }\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^{n}f(Z_i)-{\int }_{X}f\mathrm{d}\mu \right)\ge \frac{1}{n}.$$

$${\mathrm{MSE}}_Z(f):=\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^{n}f(Z_i)-{\int }_{X}f\mathrm{d}\mu \right)$$

$$\mathrm{MSE}(Z):=\underset{\begin{array}{c}f\in L^2(X,\mu )\\ \Vert f{\Vert }_2=1\end{array}}{\sup }{\mathrm{MSE}}_Z(f).$$

$$\begin{array}[]{rcl}\operatorname{MSE}_{Z}(f)&=&\displaystyle\mathbb{E}\left[\left(\frac{1}{n}\sum^{n}_{i=1}f(Z_{i})\right)^{2}\right]\\ &=&\displaystyle\mathbb{E}\left[\frac{1}{n^{2}}\sum^{n}_{i=1}f(Z_{i})^{2}+\frac{1}{n^{2}}\sum_{i\neq j}f(Z_{i})f(Z_{j})\right]\\ &=&\displaystyle\frac{1}{n^{2}}\sum^{n}_{i=1}\mathbb{E}[f(Z_{i})^{2}]+\frac{1}{n^{2}}\sum_{i\neq j}\mathbb{E}[f(Z_{i})f(Z_{j})]\\ &=&\displaystyle\frac{1}{n}+\frac{1}{n^{2}}\sum_{i\neq j}\mathbb{E}[f(Z_{i})f(Z_{j})]\\ \end{array}$$

$$f_{p,q}:=\sqrt{\frac{N}{2}}{1}_{X_p}-\sqrt{\frac{N}{2}}{1}_{X_q}.$$

$$\begin{array}[]{rcll}\operatorname{MSE}_{Z}(f_{p,q})&=&\displaystyle\frac{1}{n}&\displaystyle+\ \frac{1}{n^{2}}\sum_{i\neq j}\mathbb{E}\left[f_{p,q}(Z_{i})f_{p,q}(Z_{j})\right]\\ &=&\displaystyle\frac{1}{n}&\displaystyle+\ \frac{1}{n^{2}}\sum_{i\neq j}\left(\sum_{k}\mathbb{P}(Z_{i}\in X_{k}\ et\ Z_{j}\in X_{k})f_{p,q}(X_{k})^{2}\right.\\ &&&\displaystyle+\left.\sum_{l\neq m}\mathbb{P}(Z_{i}\in X_{l}\ et\ Z_{j}\in X_{m})f_{p,q}(X_{l})f_{p,q}(X_{m})\right)\\ &=&\displaystyle\frac{1}{n}&\displaystyle+\ \frac{1}{n^{2}}\frac{N}{2}\sum_{i\neq j}\left(\mathbb{P}(Z_{i}\in X_{p}\ et\ Z_{j}\in X_{p})\right.\\ &&&\displaystyle+\ \mathbb{P}(Z_{i}\in X_{q}\ et\ Z_{j}\in X_{q})\\ &&&\displaystyle-\ \mathbb{P}(Z_{i}\in X_{p}\ et\ Z_{j}\in X_{q})\\ &&&\displaystyle-\left.\ \mathbb{P}(Z_{i}\in X_{q}\ et\ Z_{j}\in X_{p})\right).\\ \end{array}$$

$${\mathrm{MSE}}_Z(f_{p,q})\ge \frac{1}{n}-\frac{1}{n^2}\frac{N}{2}\left(\sum _{i\ne j}\mathbb{P}(Z_i\in X_{p}et{Z}_j\in X_q)+\mathbb{P}(Z_i\in X_{q}et{Z}_j\in X_p)\right).$$

$${\theta }_{p,q}:=\sum _{i\ne j}\mathbb{P}(Z_i\in X_{p}et{Z}_j\in X_q)+\mathbb{P}(Z_i\in X_{q}et{Z}_j\in X_p).$$

$$\begin{array}[]{rcl}\displaystyle\sum_{p\neq q}\theta_{p,q}&=&2\sum_{p\neq q}\sum_{i\neq j}\mathbb{P}(Z_{i}\in X_{p}\ et\ Z_{j}\in X_{q})\\ &=&2\sum_{i\neq j}\sum_{p\neq q}\mathbb{P}(Z_{i}\in X_{p}\ et\ Z_{j}\in X_{q})\\ &=&2\sum_{i\neq j}\mathbb{P}(Z_{i}\ \mbox{et}\ Z_{j}\ \mbox{ne sont pas dans le même morceau de la partition})\\ &\leq&2n(n-1).\\ \end{array}$$

$$\begin{array}[]{rcl}\operatorname{MSE}_{Z}(f_{p,q})&\geq&\displaystyle\frac{1}{n}-\frac{1}{n^{2}}\frac{N}{2}2\frac{n(n-1)}{N(N-1)}\\ &=&\displaystyle\frac{1}{n}-\frac{n-1}{n(N-1)}\\ &=&\displaystyle\frac{1}{n}\left(1-\frac{n-1}{N-1}\right).\\ \end{array}$$

$${\mathrm{MSE}}_Z=\frac{1}{n}\left(1-\frac{n-1}{N-1}\right).$$

$$\pi :=\frac{(N-n)!}{N!}\sum _{\begin{array}{c}{i}_1,\cdots ,i_n\in X\\ \forall j\ne k,\\ i_j\ne i_k\end{array}}{\delta }_{(i_1,\cdots ,i_n)},$$

$$\begin{array}[]{rcl}\sum_{l\neq m}\mathbb{E}[f(Z_{l})f(Z_{m})]&=&\displaystyle\sum_{l\neq m}\mathbb{E}\left[\sum_{\begin{subarray}{c}A\subset X\\ |A|=n\end{subarray}}\sum_{\begin{subarray}{c}i_{1},\cdots,i_{n}\in A\\ \forall j\neq k\\ i_{j}\neq i_{k}\end{subarray}}\mathbf{1}_{\{Z_{1}=i_{1},\cdots,Z_{n}=i_{n}\}}f(Z_{l})f(Z_{m})\right]\\ &=&\displaystyle\sum_{l\neq m}\sum_{\begin{subarray}{c}A\subset X\\ |A|=n\end{subarray}}\sum_{\begin{subarray}{c}i_{1},\cdots,i_{n}\in A\\ \forall j\neq k\\ i_{j}\neq i_{k}\end{subarray}}\mathbb{P}\left[Z_{1}=i_{1},\cdots,Z_{n}=i_{n}\right]f(i_{l})f(i_{m})\\ &=&\displaystyle\dbinom{N}{n}^{-1}\sum_{\begin{subarray}{c}A\subset X\\ |A|=n\end{subarray}}\sum_{\begin{subarray}{c}p,q\in A\\ p\neq q\end{subarray}}f(p)f(q)\\ &=&\displaystyle\dbinom{N-2}{n-2}\dbinom{N}{n}^{-1}\sum_{\begin{subarray}{c}p,q\in X\\ p\neq q\end{subarray}}f(p)f(q)\\ &=&\displaystyle\frac{n(n-1)}{N(N-1)}\sum_{p\in X}f(p)\sum_{\begin{subarray}{c}q\in X\\ q\neq p\end{subarray}}f(q)\\ &=&\displaystyle\frac{n(n-1)}{N-1}\|f\|^{2}_{2}\\ &=&\displaystyle\frac{n(n-1)}{N-1}.\\ \end{array}$$

$${\mathrm{MSE}}_Z(f)=\frac{1}{n}\left(1-\frac{n-1}{N-1}\right).$$

Full text

1. Introdution and statement of the results

Let $(X,\mu)$ be a probability space. We are interested in the following general question: if $f$ is a measurable, real or complex-valued function on $X$, how can we efficiently compute the integral $\int_{X}f\mathop{}\!\mathrm{d}\mu$ ? The famous Monte-Carlo method is a solution to this problem: just choose an integer $n$ big enough, and draw $Z_{1},\cdots,Z_{n}$ independent $X$-valued random variables (that is, random points) of law $\mu$, and form the mean $\frac{1}{n}\sum^{n}_{i=1}f(Z_{i})$, called the Monte-Carlo estimator.

We measure the quality of this method by computing what we call the mean squared error: we have the well-known equality, valid for all $n\in\mathbb{N}^{*}$ and $f\in L^{2}(X,\mu)$,

[TABLE]

and we obtain the following equality, concerning the worst-case mean squared error:

[TABLE]

In this paper, we study the question of measuring the worst-case mean squared error, in the general situation where the points $Z_{i}$ are not supposed independent, and we prove the following theorem and its corollary.

Theorem**.**

Let $(X,\mu)$ be a probability space, let $N,n\in\mathbb{N}^{*}$, and $Z:=(Z_{1},\cdots,Z_{n})$ an $n$-tuple of random points on $X$ such that for all $i$, the law of $Z_{i}$ is $\mu$. We do not assume that the $Z_{i}$’s are independent. Furthermore, we assume that $X$ can be partitioned in $N$ measurable subsets of equal measure.

We then have

[TABLE]

Corollary**.**

Let $(X,\mu)$ be an atomless probability space, $n\in\mathbb{N}^{*}$, and $Z:=(Z_{1},\cdots,Z_{n})$ an $n$-tuple of random points on $X$ such that for all $i$, the law of $Z_{i}$ is $\mu$. We do not assume that the $Z_{i}$’s are independent.

We then have

[TABLE]

Remark**.**

As we shall see in the paper, in the case where $X:=\{1,\cdots,N\}$ and $\mu$ is the uniform measure on $X$, the inequality of the theorem is an equality when the law of $Z$ is the uniform measure on the set of $n$-tuples of points in $X$ such that the coordinates are pairwise different. This random $n$-tuple is then, in the sense of the worst-case mean squared error, than an independent $n$-tuple.

As we saw before, the inequality in the corollary is an equality if the $Z_{i}$’s are independent. We don’t know if this condition is necessary. It is, to our opinion, worth knowing that in [LPS86], the authors build, for all prime $p$ such that $p\equiv 1[4]$, a $(p+1)$-tuple $Z$ of uniform random points on the $2$-sphere which are not independent, and prove that its worst-case mean squared error is $\frac{4p}{(p+1)^{2}}$, which is approximately $4$ times the lower bound in the corollary. In the article [LP18], it is shown that their construction is optimal, in a broad framework.

We confess our astonishment of having found no trace of these statements, which answer a question that we find both natural and general, and in an elementary way.

Acknowledgements**.**

We woud like to thank Sébastien Darses, Thibault Espinasse, Alexandre Gaudillière, Pierre Mathieu, Clothilde Melot, Pierre Pudlo and more particularly Christophe Pittet for the conversations that we had about the questions studied in this paper, and for their encouragement.

2. Proofs

To alleviate the presentation, we use the following notation: we consider the numbers

[TABLE]

et

[TABLE]

First of all, if $f\in L^{2}(X,\mu)$, we notice that $\operatorname{MSE}_{Z}(f)=\operatorname{MSE}(Z)\left(f-\int_{X}f\mathop{}\!\mathrm{d}\mu\right)$. Consequently, $\operatorname{MSE}(Z)$ is also the $\sup$ of the $\operatorname{MSE}_{Z}(f)$ for $f$ of norm $1$ zero integral.

Let $f\in L^{2}(X,\mu)$, of norm $1$ and zero integral. We have that

[TABLE]

and we recover the fact recalled above: if the $Z_{i}$’s are pairwise independent, and if $f$ is of norm $1$ of zero integral, $\operatorname{MSE}_{Z}(f)=\frac{1}{n}$.

Let us prove the theorem.

Proof of the theorem.

Let $X_{1},...,X_{N}$ be measurable subsets that partition $X$, all of measure $\frac{1}{N}$, with $N\geq 2$. Let us denote, for $p\in\{1,...,N\}$, $\mu_{p}:=\mu(X_{p})$. For every $(p,q)\in\{1,...,N\}^{2}$, we set

[TABLE]

Moreover, we will denote, for $k\in\{1,...,N\}$, $f_{p,q}(X_{k})$ the value that $f_{p,q}$ takes on $X_{k}$ - this abuse of notation is harmless because $f_{p,q}$ is constant on the $X_{i}$’s.

$f_{p,q}$ is visibly of zero integral, and if $p\neq q$, its norm is $1$.

We will prove that there are different $p,q\in\{1,...,N\}$ such that $\operatorname{MSE}_{Z}(f_{p,q})\geq\frac{1}{n}\left(1-\frac{n-1}{N-1}\right)$.

Let $p,q\in\{1,...,N\}$. We have that

[TABLE]

from which we deduce the inequality

[TABLE]

Let us denote

[TABLE]

Let us compute:

[TABLE]

Now, since this sum of $N(N-1)$ numbers is lower or equal than $2n(n-1)$, then one of the terms must be lower or equal than $2\frac{n(n-1)}{N(N-1)}$. For a couple $(p,q)$ such that $\theta_{p,q}\leq 2\frac{n(n-1)}{N(N-1)}$, we then have

[TABLE]

∎

Here’s an example where the inequality is an equality.

Proposition**.**

If $X:=\{1,\cdots,N\}$, if $\mu$ is the uniform probability on $X$, if $n\leq N$, and if the law of $Z$ is the uniform measure on the set of $n$-tuples of points in $X$ which coordinates are pairwise different, then the inequality in the theorem is an equality, that is,

[TABLE]

Proof.

Let $\pi$ be the measure on $X^{n}$ defined by

[TABLE]

where $\delta$ is the notation for a Dirac measure. In words, $\pi$ is the uniform measure on the set of $n$-tuples of points in $X$ which coordinates are pairwise different. Let $Z=(Z_{1},\cdots,Z_{n})$ be an $n$-tuple of law $\pi$ (we then have, for all $i$, that $Z_{i}$ is uniform on $X$).

Let $f\in L^{2}(X,\mu)$ be of norm $1$, and such that $\int f\mathop{}\!\mathrm{d}\mu=0$. Let us compute:

[TABLE]

We therefore have

[TABLE]

∎

Let us prove the corollary.

Proof of the corollary.

We will prove that for all $\epsilon>0$, we have that $\operatorname{MSE}(Z)\geq\frac{1}{n}-\epsilon$, which is enough. According to a theorem of Sierpiński [Sie22], every atomless probability space is such that for every $a\in[0,1]$, there is a measurable subset of $X$ of measure $a$. From this, it is easy, for all arbitrarily big $N$, to partition $X$ in $N$ of measurable subsets of equal measure. If we choose $N$ such that $\frac{1}{n}\left(1-\frac{n-1}{N-1}\right)\geq\frac{1}{n}-\epsilon$, which is obviously possible, then according to the theorem, it is possible to find $f$ of norm $1$, zero integral, such that $\operatorname{MSE}_{Z}(f)\geq\frac{1}{n}-\epsilon$. ∎

For the sake of completeness, we add a simple proof of Sierpiński’s theorem.

Complement** (Sierpiński’s theorem on atomless probability spaces).**

If $(X,\mathscr{B},\mu)$ is an atomless probability space, then for every measurable $A\subset X$, there exists $\phi:[0,\mu(A)]\rightarrow\mathscr{B}$ non-decreasing, such that $\forall t\in[0,\mu(A)],\quad\mu(\phi(t))=t$.

Proof.

The hypothesis of $X$ being atomless means that for every measurable $B\subset X$ such that $\mu(B)>0$, there exists a measurable $C\subset B$ such that $0<\mu(C)<\mu(B)$.

Let $A$ be a measurable subset of $X$, such that $\mu(A)>0$ (if $\mu(A)=0$, it is enough to define $\phi(0):=A$). By applying Zorn’s lemma, we obtain a $\phi:I\rightarrow\mathscr{B}$ where $I$ is a subset of $[0,\mu(A)]$, $\phi$ is non-decreasing, such that $\forall i\in I$, $\mu(\phi(i))=i$, and such that $\phi$ has no strict extension that satisfies these properties. Let us show that $I$ equals $[0,\mu(A)]$.

On the one hand, $I$ is closed. Indeed, let $(x_{n})_{n\in\mathbb{N}}$ be a sequence of elements in $I$ that converges to some $x$. Let us show that $x\in I$. We can assume, up to extracting a subsequence, that $(x_{n})_{n}$ is monotonous. If $x\not\in I$, let us define $\tilde{\phi}:=I\cup\{x\}\rightarrow\mathscr{B}$ that extends $\phi$ by defining $\tilde{\phi}(x):=\bigcap_{n}\phi(x_{n})$ if $(x_{n})_{n}$ is non-increasing, and $\tilde{\phi}(x):=\bigcup_{n}\phi(x_{n})$ if $(x_{n})_{n}$ non-decreasing. According to $\mu$’s continuity properties, $\mu(\phi(x))=\lim_{n}x_{n}=x$, and according to the monotony properties of $\mu$, $\tilde{\phi}$ is non-decreasing. $\tilde{\phi}$ is therefore a strict extension of $\phi$ that verifies the same properties. This is a contradiction. So $x\in I$, and therefore, $I$ is closed.

On the other hand, $I$ verifies $\forall a,b\in I,\ a<b\Rightarrow\left(\exists c\in I,\ a<c<b\right)$ (we say that $I$ is order-dense). Indeed, if there are $a,b\in I$ such that $a<b$ and $]a,b[\cap I=\emptyset$, then let us use the hypothesis that $X$ is atomless, which provides a measurable $C\subset\phi(b)\setminus\phi(a)$ such that $0<\mu(C)<b-a$. Let us then define $\tilde{\phi}:I\cup\{a+\mu(C)\}$ that extends $\phi$ by defining $\tilde{\phi}(a+\mu(C)):=\phi(a)\cup C$. Then $\mu(\tilde{\phi})(a+\mu(C))=\mu(\phi(a)\cup C)=a+\mu(C)$. According to $\mu$’s monotony properties, $\tilde{\phi}$ est non-decreasing. $\tilde{\phi}$ is then a strict extension of $\phi$ that verifies the same properties. This is a contradiction. Therefore, $I$ is order-dense.

So $I$ is closed and order-dense. Therefore, $I=[0,\mu(A)]$. ∎