Multi-Scale Vector Quantization with Reconstruction Trees

TL;DR

This paper introduces a multi-scale vector quantization method called reconstruction trees, which efficiently reconstructs data by leveraging a family of partitions, with theoretical analysis under various data distribution assumptions.

Contribution

It presents a novel multi-scale vector quantization algorithm inspired by decision trees, with theoretical analysis of its expected distortion for data from unknown distributions.

Findings

Achieves low expected distortion under regularity assumptions.

Provides asymptotic and finite sample bounds.

Applicable to data on Riemannian manifolds.

Abstract

We propose and study a multi-scale approach to vector quantization. We develop an algorithm, dubbed reconstruction trees, inspired by decision trees. Here the objective is parsimonious reconstruction of unsupervised data, rather than classification. Contrasted to more standard vector quantization methods, such as K-means, the proposed approach leverages a family of given partitions, to quickly explore the data in a coarse to fine-- multi-scale-- fashion. Our main technical contribution is an analysis of the expected distortion achieved by the proposed algorithm, when the data are assumed to be sampled from a fixed unknown distribution. In this context, we derive both asymptotic and finite sample results under suitable regularity assumptions on the distribution. As a special case, we consider the setting where the data generating distribution is supported on a compact Riemannian sub-manifold. Tools from differential geometry and concentration of measure are useful in our analysis.

Equations

$$P_{\Lambda }(x)=\sum _{j=1}^k{c}_j\text{mathbbm}{1}_{I_j}(x)x\in {\mathbb{R}}^D,$$

$$\text{mathbbm}{1}_I(x)=\{\begin{array}{ll}1& x\in I\\ 0& x\notin I\end{array}.$$

$$\hat{\mathcal{E}}[P_{\Lambda }]=\frac{1}{n}\sum _{i=1}^n{\Vert x_i-P_{\Lambda }(x_i)\Vert }^2,$$

$$\mathcal{E}[P_{\Lambda }]=\mathbb{E}[{\Vert X-P_{\Lambda }(X)\Vert }^2].$$

$$\{\begin{array}{l}\mathcal{X}={\bigcup }_{I\in {\Lambda }_j}I\\ I\cap J=\varnothing \forall I,J\in {\Lambda }_j,I\ne J\\ ♯{\Lambda }_j<+\infty \\ I\in \mathcal{B}(\mathcal{X})I\in {\Lambda }_j\end{array};$$

$$\{\begin{array}{l}I=\bigcup _{J\in \mathcal{C}(I)}J\\ ♯\mathcal{C}(I)\le a\end{array},$$

$$\Lambda (\mathcal{T})=\{I\in \mathbb{T}:I\notin \mathcal{T},\mathcal{P}(I)\in \mathcal{T}\},$$

$$n_I=\sum _{i=1}^n\text{mathbbm}{1}_I(x_i),{\hat{c}}_I=\{\begin{array}{ll}\frac{1}{n_I}\sum _{i=1}^n{x}_i\text{mathbbm}{1}_I(x_i)& \text{if }x\in I\text{ and }{n}_I\ne 0\\ {\hat{x}}_I^{\ast }& \text{if }x\in I\text{ and }{n}_I=0\end{array}.$$

$${\hat{\mathcal{E}}}_I=\frac{1}{n}\sum _{x_i\in I}{\Vert x_i-{\hat{c}}_I\Vert }^2=\frac{1}{n}\sum _{i=1}^n{\Vert x_i-{\hat{c}}_I\Vert }^2\text{mathbbm}{1}_I(x_i),$$

$${\widehat{ϵ}}_I^2={\hat{\mathcal{E}}}_I-\sum _{J\in \mathcal{C}(I)}{\hat{\mathcal{E}}}_J=\frac{1}{n}\sum _{J\in \mathcal{C}(I)}{\Vert {\hat{c}}_J-{\hat{c}}_I\Vert }^2,$$

$$j_n=\lfloor \frac{\gamma \ln n}{\ln a}\rfloor ⟹a^{j_n}\le n^{\gamma }.$$

$${\hat{\mathcal{T}}}_{\eta }=\{\begin{array}{ll}\{\mathcal{X}\}& \text{ if }{\widehat{ϵ}}_I<\eta \forall I\in \bigcup _{j\le j_n}{\Lambda }_j\\ \{I\in \mathbb{T}\mid \exists j\le j_n,J\in {\Lambda }_j,J\subset I,{\widehat{ϵ}}_J\ge \eta \}& \text{otherwise}\end{array}$$

$${\hat{P}}_{\eta }=\sum _{I\in {\hat{\Lambda }}_{\eta }}{\hat{c}}_I\text{mathbbm}{1}_I(x){\hat{\Lambda }}_{\eta }=\Lambda ({\hat{\mathcal{T}}}_{\eta }).$$

$$\underset{F\in \mathcal{H}}{\min }\hat{\mathcal{E}}[F],$$

$$V_j=\{x\in {\mathbb{R}}^D|\Vert x-c_j\Vert \le \Vert x-c_i\Vert ,\forall i=1,\dots ,k,i\ne j\}.$$

$$\underset{c_1,\dots ,c_k}{\min }\frac{1}{n}\sum _{i=1}^n\underset{j=1,\dots ,k}{\min }{\Vert x_i-c_j\Vert }^2.$$

$$\mathrm{diam}(\mathcal{X})=\underset{x,y\in \mathcal{X}}{\sup }\Vert x-y\Vert \le 1.$$

$${\mathrm{diam}}_{\rho }(I)$$

$$\mathrm{diam}(I)\lesssim b^{-j_I}\text{ for all }I\in \mathbb{T},$$

$$\rho (I)\lesssim c^{-j_I}\text{ for all }I\in \mathbb{T},$$

$$\rho (E)=\underset{E}{\int }p(x)dxE\in \mathcal{B}(\mathcal{X}),$$

$${\Lambda }_j=\{[2^{-j}(k_1-1),2^{-j}{k}_1)\times \dots \times [2^{-j}(k_D-1),2^{-j}{k}_D)\mid k_1,\dots ,k_D=1,...,\dots ,2^j\}j\in \mathbb{N}$$

$$\rho (E)=\underset{E\cap \mathcal{M}}{\int }p(x)d{\rho }_{\mathcal{M}}(x)$$

$$0<p_1\le p(x)\le p_2<+\infty$$

$$\text{mathds}P\left[\mathcal{E}[{\hat{P}}_{\eta }]\gtrsim {\eta }^{\frac{4\sigma }{2\sigma +1}}(1+t)\right]\lesssim {\eta }^{-\frac{2}{2\sigma +1}}\exp \left(-c_{\mathbb{T}}n{\eta }^{2}t\right)+(n^{\gamma }+{\eta }^{-\frac{2}{2\sigma +1}})\exp \left(-c_{a}n{\eta }^2\right)t>0,$$

$${\eta }_n=\sqrt{\frac{(\gamma +\beta )\ln n}{c_{a}n}}\text{and}{\hat{P}}_n={\hat{P}}_{{\eta }_n}n\ge 1,$$

$$\text{mathds}P\left[\mathcal{E}[{\hat{P}}_n]\gtrsim {\left(\frac{\ln n}{n}\right)}^{\frac{2\sigma }{2\sigma +1}}(1+t)\right]\lesssim \frac{1}{n^{\beta }}+\frac{1}{n^{{\overline{c}}_{\mathbb{T}}t-1}},$$

$$\underset{t\to +\infty }{\lim }\underset{n\to +\infty }{\mathrm{limsup}}\underset{\rho \in {\mathcal{P}}_{b,s}(\mathbb{T})}{\sup }\text{mathds}P\left[\mathcal{E}[{\hat{P}}_n]\gtrsim {\left(\frac{\ln n}{n}\right)}^{\frac{2\sigma }{2\sigma +1}}t\right]=0,$$

$$\text{mathds}P\left[\mathcal{E}[{\hat{P}}_n]\ge c_1{\left(\frac{\ln n}{n}\right)}^{\frac{2\sigma }{2\sigma +1}}\right]\le c_2\frac{1}{n^{\beta }}$$

$${\rho }_I=\rho (I),$$

Full text

1 Introduction

Dealing with large high-dimensional data-sets is a hallmark of modern signal processing and machine learning. In this context, finding parsimonious representation from unlabeled data is often key to both reliable estimation and efficient computations, and more generally for exploratory data analysis. A classical approach to this problem is principal component analysis (PCA), relying on the assumption that data are well represented by a linear subspace. Starting from PCA a number of developments can be considered to relax the linearity assumption. For example kernel PCA is based on performing PCA after a suitable nonlinear embedding [26]. Sparse dictionary learning tries to find a set of vectors on which the data can be written as sparse linear combinations [21]. Another line of works assumes the data to be sampled from a distribution supported on a manifold and includes isomap [29], Hessian eigenmaps [8], Laplacian eigenmaps [3] and related developments such as diffusion maps [20]. A more recent and original perspective has been proposed in [14], and called geometric multi-resolution analysis (GMRA). Here the idea is to borrow and generalize ideas from multi-resolution analysis and wavelet theory [17], to derive locally linear representation organized in a multi-scale fashion. The corresponding algorithm is based on a cascade of local PCAs and is reminding of classical decision trees for function approximation, see e.g. [12]. In this paper we further explore the ideas introduced to GMRA which is our main reference.

Indeed, we consider these ideas in the context of vector quantization, which is another classical, and extreme, example of parsimonious representation. Here, a set of centers and corresponding partition is considered, and then all data points in each cell of the partition are represented by the corresponding center. The most classical approach in this context is probably $k$-means, where a set of centers (means) is defined by a non-convex optimization problem over all possible partitions. Our approach offers an alternative to $k$-means, by following the basic idea of GMRA and decision trees, but considering local means rather than local PCA. In this view, our approach can be seen as a zero-th order version of GMRA, hence providing a piece-wise constant data approximation. Compared to $k$-means, the search for a partition is performed through a coarse-to-fine recursive procedure, rather than by global optimization. A strategy that we call reconstruction tree. As a byproduct the corresponding vector quantization is multi-scale, and naturally yields a multi-resolution representation of the data. Our main technical contribution is a theoretical analysis of the above multi-scale vector quantization procedure. We consider a statistical learning framework, where the data are assumed to be sampled according to some fixed unknown distribution and measure performance according to the so called expected distortion, measuring the reconstruction error with respect to the whole data distribution. Our main result is deriving corresponding finite sample bound in terms of natural geometric assumptions.

The rest of the paper is organized as follows. After describing the basic ideas in the context of vector quantization in Section 2, we present the algorithm we study in Section 3. In Section 4, we introduce the basic theoretical assumptions needed in our analysis, and illustrate them considering the case where the data are sampled from a manifold. In Section 5, we present and discuss our main results, and detail the main steps in their proofs. All other proofs are deferred to the Appendix.

2 Vector quantization & distortion

We next introduce the problem of interest and comment on its connections with related questions.

A vector quantization (VQ) procedure is defined by a set of code vectors/centers and an associated partition of the data space. The idea is that compression can be achieved replacing all points in a given cell of the partition by the corresponding code vector. More precisely, assuming that the data space is $\mathbb{R}^{D}$, consider a set of code vectors $c_{1}\in I_{1},\dots,c_{k}\in I_{k},$, where the set of cells $\Lambda=\{I_{1},\dots,I_{k}\}$, defines a partition of $\mathbb{R}^{D}$. A nonlinear projection $P_{\Lambda}:\mathbb{R}^{D}\to\mathbb{R}^{D}$ can be defined by

[TABLE]

where $\mathbbm{1}_{I}$ is the characteristic function of $I$, * i.e.*

[TABLE]

Given a set of points $x_{1},\dots,x_{n}$ the error (distortion) incurred by this nonlinear projection can be defined as

[TABLE]

where $\left\lVert\cdot\right\rVert$ is the Euclidean norm in $\mathbb{R}^{D}$. If we consider the data to be identical and independent samples of a random variable $X$ in $\mathbb{R}^{D}$, then the following error measure can also be considered

[TABLE]

The above error measure is the expected distortion associated to the quantization defined by $P_{\Lambda}$.

In the following we are interested in deriving VQ schemes with small expected distortion given a dataset $x_{1},\dots,x_{n}$ of $n$ samples of $X$. Before describing the algorithm we propose, we add two remarks.

Remark 1** (Comparison to supervised learning).**

Classical supervised learning is concerned with the problem of inferring a functional relationship $f$ given a set of input-output pairs $(x_{1},y_{1}),\dots,(x_{n},y_{n})$. A classical error measure is the least squares loss $\left\lVert y-f(x)\right\rVert^{2}$ (if the outputs are vectors valued). A parallel between the above setting and supervised learning can be seen, considering the case where the input and output spaces coincide and the least squares loss would be $\left\lVert x-f(x)\right\rVert^{2}$. Clearly, in this case an optimal solution is given by the identity map, unless further constraints are imposed. Following the above remark, we can view the nonlinear projection $P_{\Lambda}$ as a piece-wise constant approximation of the identity map, possibly providing a parsimonious representation.

Remark 2** (Vector quantization as Dictionary Learning).**

A dictionary is a set of vectors (called atoms) $a_{1},\dots,a_{p}$ that can be used to approximately decompose each point $x$ in the data space, i.e. $x\approx\sum_{j=1}^{p}a_{i}\beta_{j},$ with $\beta=(\beta_{1},\dots,\beta_{p})\in\mathbb{R}^{p}$ being a coefficients vector. Given a set of points $x_{1},\dots,x_{n}$, dictionary learning is the problem of estimating a dictionary, as well as a set of coefficients vectors.Vector quantization can be viewed as a form of dictionary learning where the code vectors are the atoms, and coefficients vectors are binary and have at most one non zero component [21].

The above remarks provide different views on the problem under study. We next discuss how ideas from decision trees in supervised learning can be borrowed to define a novel VQ approach.

3 Multi-scale vector quantization via reconstruction trees

We next describe our approach to Multi-Scale Vector Quantization (MSVQ), based on a recursive procedure that we call reconstruction trees, since it is inspired by decision trees for function approximation. The key ingredient in the proposed approach is a family of partitions organized in a tree. The partition at the root of the tree has the largest cells, while partitions with cells of decreasing size are found in lower leafs. This * partition tree* provides a multi-scale description of the data space: the lower the leaves, the finer the scale. The idea is to use data to identify a subset of cells, and corresponding partition, providing a VQ solution (1) with low expected distortion (2). We next describe this idea in detail.

3.1 Partition trees and subtrees

We begin introducing the definition of a partition tree. In the following we denote by $\mathcal{X}\subset\mathbb{R}^{D}$ the data space endowed with its natural Borel $\sigma$-algebra $\mathcal{B}(\mathcal{X})$ and by $\sharp A$ the cardinality of a set $A$.

Definition 3**.**

A partition tree $\mathbb{T}$ is a denumerable family $\{\Lambda_{j}\}_{j\in\mathbb{N}}$ of partitions of $\mathcal{X}$ satisfying

1. a)

$\Lambda_{0}=\{\mathcal{X}\}$* is the root of the tree;* 2. b)

each family $\Lambda_{j}=\{I\}_{I\in\Lambda_{j}}$ is a finite partition of $\mathcal{X}$ of Borel subsets, i.e

[TABLE] 3. c)

for each $I\in\Lambda_{j}$, there exists a family $\mathcal{C}(I)\subseteq\Lambda_{j+1}$ such that

[TABLE]

where $a\in(0,+\infty)$ is a constant depending only on $\mathbb{T}$.

Note that, we allow the partition tree to have arbitrary, possibly infinite, depth, needed to derive asymptotic results.

Further, notice that, since $\#\Lambda_{j}\leq a^{j}$ the constant $a$ characterizes how the cardinality of each partition increases at finer scales. The case $a=2$ corresponds to dyadic trees.

We add some further definitions. For any $j\in\mathbb{N}$, and $I\in\Lambda_{j}$ the depth of $I$ is $j$ and is denoted by $j_{I}$. The cells in $\mathcal{C}(I)\subset\Lambda_{j+1}$ are the children of $I$, the unique cell $J\in\Lambda_{j-1}$ such that $I\in\mathcal{C}(J)$ is the parent of $I$ and is denoted by $\mathcal{P}(I)$ (by definition $\mathcal{P}(\mathcal{X})=\mathcal{X}$). We regard $\mathbb{T}$ as a set of nodes where each node is defined by a cell $I$ with its parent $\mathcal{P}(I)$ and its children $\mathcal{C}(I)$. The following definition will be crucial.

Definition 4**.**

A (proper) subtree of $\mathbb{T}$ is a family $\mathcal{T}\subset\mathbb{T}$ of cells such that $\mathcal{P}(I)\in\mathcal{T}$ for all $I\in\mathcal{T}$ and

[TABLE]

denotes the set of outer leaves.

It is important in what follows that $\Lambda(\mathcal{T})$ is a partition of $\mathcal{X}$ if $\mathcal{T}$ is finite, see Lemma  11.

3.2 Reconstruction trees

We next discuss a data driven procedure to derive a suitable partition and a corresponding nonlinear projection. To do this end, we need a few definitions depending on an available dataset $x_{1},\dots,x_{n}$.

For each cell $I$, we fix an arbitrary point $\hat{x}^{*}_{I}\in\mathcal{X}$ and define the corresponding cardinality and center of mass, respectively, as

[TABLE]

If $0\in\mathcal{X}$, a typical choice is $\hat{x}_{I}^{*}=0$ for all cells $I\in\mathbb{T}$. While $\mathcal{E}(\hat{P}_{n})$ depends on the choice of $\hat{x}_{I}^{*}$, our bounds hold true for all choices. We point out that it is more convenient to choose $\hat{x}_{I}^{*}\in I$, as this (arbitrary) choice produces an improvement of $\mathcal{E}(\hat{P}_{n})$ for free, in particular whenever $\mathbb{E}[X\in I]>0$ but $n_{I}=0$.

Using this quantity we can define a local error measure for each cell $I$,

[TABLE]

as well as the potential error difference induced by considering a refinement,

[TABLE]

where the second equality is consequence of the between-within decomposition of the variance. Following [4], we first truncate the partition tree at a given depth, depending on the size of the data set. More precisely, given $\gamma>0$, we set

[TABLE]

Deeper trees are considered as data size grows.

As a second step, we fix a threshold $\eta>0$ and select the cells such that $\hat{\epsilon}_{I}\geq\eta$. Since $\hat{\epsilon}_{I}$ is not an decreasing function with the depth of the tree, this requires some care – see Remark 10 for an alternative construction. Indeed, we define the $\eta$-dependent subtree

[TABLE]

and $\widehat{\Lambda}_{\eta}$ is defined as outerleaves of $\widehat{\mathcal{T}}_{\eta}$, * i.e.* $\widehat{\Lambda}_{\eta}=\Lambda(\widehat{\mathcal{T}}_{\eta})$. Note that $\widehat{\mathcal{T}}_{\eta}$ is finite, so that by Lemma 11 $\widehat{\mathcal{T}}_{\eta}$ is a partition of $\mathcal{X}$ such that $j_{I}\leq(\gamma\ln n)/\ln a$ for all $I\in\widehat{\Lambda}_{\eta}$.

The code vectors are the centers of mass of the cells the above empirical partition, and the corresponding nonlinear projection is

[TABLE]

We add a a few comments, the above vector quantization procedure, that we call reconstruction tree, is recursive and depends on the threshold $\eta$. Different quantizations and corresponding distortions are achieved by different choices of $\eta$. Smaller values of $\eta$ correspond to vector quantization quantizations with smaller distortion. It is a clear that the empirical distortion becomes zero for a suitably small $\eta$ corresponding to having a single point in each cell. Understanding the behaviour of the expected distortion as function of $\eta$ and the number of points is our main theoretical contribution. Before discussing these results we discuss the connection of the above approach to related ideas. A similar construction is given in [14], where however the thresholding criterion $\eta$ depends on the scale, see Section 2.3 of the cited reference.

3.3 Comparison with related Topics

The above approach can be compared with a number of different ideas.

Decision and Regression Trees

We call the above procedure Reconstruction Tree, since its definition is formally analogous to that of decision trees for supervised learning; see for example [13], Chapter 8. In particular our construction and analysis follows closely that of tree based estimators studied in [4], in the context of least square regression. As commented in Remark 1 our problem can actually be interpreted as a special instance of vector valued regression where the regression function $f_{\rho}$ is the identity function from the input space to itself (regarded as the output space): indeed, referring to the notation of [4], the conditional distribution of the output $y$ given $x$ is the Delta measure at $x$. From this point on the two formalisms overlap, in that $\left\lVert f-f_{\rho}\right\rVert_{L_{2}}^{2}$ becomes $\mathbb{E}[\left\lVert X-f(X)\right\rVert^{2}]$. Since the tree based estimators $f$ considered in [4] are piece-wise constant, the expected square loss cannot vanish and its analysis is non trivial. Despite the formal similarity, the two settings do exhibit distinct features. For example, the analysis in [4] is specifically formulated for scalar functions, while our analysis is necessarily vectorial in nature. In [4] a uniform bound $|y|<M$ is imposed, while in our setting we can assume a local bound for free; namely, if $f$ is constant on a cell $I\subset\mathcal{X}$ then $\left\lVert x-f(x)\right\rVert^{2}\leq{\operatorname{diam}(I)}^{2}$ for all $x\in I$. The present setting finds a natural instance in the case of a probability measure supported on a smooth manifold isometrically embedded in $\mathcal{X}$ (see Section 4), while this case is hardly addressed explicitly in the literature about least square regression. For example, the manifold case is actually discussed, in the context of classification through Decision Trees, in [28]. One last point is that the present work contains explicit quantitative results about the approximation error, see Section 5.1, while similar results are not available in the setting of [4], that aspect being typically addressed indirectly in the corresponding literature.

Empirical risk minimization

Again in analogy to supervised learning, as in [4], one can consider the minimization problem:

[TABLE]

where $\mathcal{H}$ is the (finite-dimensional) vector space of the vector fields $F:\mathcal{X}\to\mathbb{R}^{D}$, which are piecewise constant on a given partition $\Lambda$. There correspond a number of independent minimization problems, one for each cell in $\Lambda$, so that $\widehat{P}_{\eta}$ from (8) is easily shown to be a minimizer. The minimizer is not unique, since the value of $F$ is irrelevant on cells $I\in\Lambda$ such that $n_{I}=0$. Similar considerations hold for $\min_{F\in\mathcal{H}}\mathcal{E}[F]$ as well, in which case the value of $F$ is irrelevant whenever $\mathbb{E}[X\in I]=0$. See also Section 3.2, Section 5.1 and Lemma 21.

One could consider minimization over a wider class of functions, piece-wise constant on different partitions, for example on all the partitions with a given number of cells that are induced by proper subtrees $\mathcal{T}$ of a given partition tree $\mathbb{T}$. This would be a combinatorial optimization problem. The algorithm defined in (7) overcomes this issue by providing a one-parameter coarse-to-fine class of partitions, such that each refinement carries local improvements $\widehat{\epsilon}$ that are uniformly bounded. As observed in [4], such a strategy is inspired by wavelet thresholding.

Geometric multi-resolution analysis (GMRA)

A main motivation for our work is the algorithm GMRA [1, 16, 14], which introduces the idea of learning multi-scale dictionaries by geometric approximation. The main difference between GMRA and Regression Trees is that the former represents data through a piece-wise linear approximation, while the latter through a piece-wise constant approximation. More precisely, rather than considering the center of mass of the data in each cells (4), a linear approximation is obtained by (local) Principal Component Analysis, so that the data belonging to a cell are sent to a linear subspace of suitable dimension, the latter approach being particularly natural in the case of data supported on a manifold. Another difference is in the thresholding strategy: unlike [4] and our work, in [14] the local improvement $\widehat{\epsilon}$ is scaled depending on its depth in the tree. One of our purposes is to check if these simpler choices affect the learning rates significantly. We provide more quantitative comparisons later in Section 5.

Wavelets

A main motivation for GMRA is extending ideas from wavelets and multi-resolution analysis to the context of machine learning, where, given the potential high dimensionality, non-regular partitions need be considered; this point is discussed in [1, 16, 14] and references therein. Indeed partition trees generalize the classic notion of dyadic partitions. In this view, given the piece-wise constant nature of reconstruction trees, a parallel can be drawn between the latter and classical Haar wavelets.

$k$-Means

Our procedure being substantially a vector quantization algorithm, a comparison with the most common approach to vector quantization, namely $k$-means, is in order. In $k$-means, a set of $k$ code vectors $c_{1},\dots,c_{k}$ are derived from the data and used to define corresponding partitions via the corresponding Voronoi diagram

[TABLE]

Code vectors are defined by the minimization of the following empirical objective

[TABLE]

This minimization problem is non convex and is typically solved by alternating minimization, a procedure referred to as Lloyd’s algorithm [15]. The inner iteration assigns each point to a center, hence a corresponding Voronoi cell. The output minimization can be easily shown to update the code vectors by computing the center of mass, * the mean*, of each Voronoi cell. In general the algorithm is ensures to decrease or at least not increase the objective function and to converge in finite time to a local minimum. Clearly, the initialization is important, and initializations exist yielding some stronger convergence guarantees. In particular, $k$-means++ is a random initialization providing on average an $\epsilon$-approximation to the global minimum [2].

Compared to $k$-means, reconstruction trees restrict the search for a partition over a prescribed family defined by the partition tree. In turns, they allow a fast multi-scale exploration of the data, while $k$-means requires solving a new optimization problem each time $k$ is changed. Indeed it can be shown that a solution for the $(k-1)$-means problem leads to a bad initialization for the $k$-means problem. In other words, unlike restriction trees, the partitions found by $k$-means at different scales (different values of $k$) are generally unrelated, and cannot be seen as refinements of one another.

Hierarchical clustering

Lastly, our coarse-to-fine approach can be compared with hierarchical clustering, in particular [with the so called Ward’s method, which proceeds in the opposite way. Indeed, this algorithm produces a coarser partition of the data starting from a finer. It starts with a Voronoi partition having all the data as centers, and at each step it merges a couple of cells that have the smallest so called between cluster inertia [30]. Interestingly this definition has an analogue in our algorithm. Our $\widehat{\mathcal{E}}_{I}$ corresponds to the within cluster inertia of a cell $I$ while $\widehat{\epsilon}^{2}_{I}$ to the between cluster inertia (up to a factor $1/n$) of cells that merge into $I$. Nevertheless the obtained partitions will not in general coincide, unless very specific choices are made ad hoc.

4 General assumptions and manifold setting

In this section, we introduce our main assumptions and then discuss a motivating example where data are sampled at random from a manifold.

We consider a statistical learning framework, in the sense that we assume the data to be random samples from an underlying probability measure. More precisely, we assume the available data to be a realization of $n$ identical and independent random vectors $X_{1},\ldots,X_{n}$ taking values in a bounded subset $\mathcal{X}\subset\mathbb{R}^{D}$ and we denote by $\rho$ the common law. Up to a rescaling and a translation, we assume that $0\in\mathcal{X}$ and

[TABLE]

Our main assumption relates the distribution underlying the data to the partition tree to be used to derive a MSVQ via reconstruction trees. To state it, we recall the notion of essential diamater of a cell $I$, namely

[TABLE]

Assumption 5**.**

There exists $s>0$ and $b>1$ such that for all $I\in\mathbb{T}$

[TABLE]

where $C_{1}>0$ and $C_{2}>0$ are fixed constants depending only on $\mathbb{T}$.

To simplify the notation, we write $c_{\mathbb{T}}$ for a constant depending only on $s,b,C_{1},C_{2}$ and we write $A\lesssim B$ if there exists a constant $c_{\mathbb{T}}>0$ such that $A\leq c_{\mathbb{T}}B$.

Given the partition tree $\mathbb{T}$, the parameters $s$ and $b$ define a class $\mathcal{P}_{b,s}(\mathbb{T})$ of probability measures $\rho$ and for this class we are able to provide a finite sample bound on the distortion error of our estimator $\hat{P}_{\eta}$, see (14). In the context of supervised machine learning $\mathcal{P}_{b,s}(\mathbb{T})$ is an a-priori class of distributions defining a upper learning rate, see (16a). It is an open problem to provide a lower min-max learning rate.

Clearly, (10b) is implied by the distribution-independent assumption

[TABLE]

i.e. the diameter of the cells goes to zero exponentially with their depth. This assumption ensures that the reconstruction error goes to zero and, in supervised learning, it corresponds to the assumption that the hypotheses space is rich enough to approximate any regression function, compare with condition (A4) in [14].

Eq. (10a) is a sort of regularity condition on the shape of the cells and, if it holds true, (10b) is implied by the following condition

[TABLE]

which states that the volume of the cells goes to zero exponentially with their depth.

In [14], following ideas from [4], it is introduced a suitable model class, see Definition 5, in terms of the decay of the approximation error, compare Eq. (7) of [14] with (19) below. This important point is further discussed in Section 5.1.

In many cases the parameter $s$ is related to the intrinsic dimension of the data. For example, if $\mathcal{X}=[0,1)^{D}$ is the unit cube and $\rho$ is given by

[TABLE]

where $dx$ is the Lebesgue measure of $\mathbb{R}^{D}$ and the density $p$ is bounded from above and away from zero, see (13b) below, it is easily to check that the family $\mathbb{T}=\{\Lambda_{j}\}$ of dyadic cubes

[TABLE]

is a partition tree satisfying Assumption 5 with $s=1/D$ and a suitable $b>1$. The construction of dyadic cubes can be extended to more general settings, see [7, 9] and references therein, by providing a large class of other examples, as shown by the following result. The proof is deferred to Section 6.

Proposition 6**.**

Assume that the support $\mathcal{M}$ of $\rho$ is a connected submanifold of $\mathbb{R}^{D}$ and the distribution $\rho$ is given by

[TABLE]

where $\rho_{\mathcal{M}}$ is the Riemannian volume element of $\mathcal{M}$, then there exists a partition tree $\mathbb{T}$ of $\mathcal{X}$ satisfying Assumption 5 with $s=1/d$, where $d$ is the intrinsic dimension of $\mathcal{M}$.

We recall that, as a submanifold of $\mathbb{R}^{D}$, $\mathcal{M}$ becomes a compact Riemannian manifold with Riemannian distance $d_{\mathcal{M}}$ and Riemannian volume element $\rho_{\mathcal{M}}$. We stress that the construction of the dyadic cubes only depend on $d_{\mathcal{M}}$. Proposition 6 has to be compared with Proposition 3 and Lemma 6 in [14].

By inspecting the proof of the above result, it is possible to show that a partition tree satisfying Assumptions 5 always exists if there are a metric $d$ and a Borel measure $\nu$ on $\mathcal{M}$ such that $(\mathcal{M},d,\nu)$ is an Ahlfors regular metric measure [10, p. 413], clg$\rho$ has density $p$ with respect to $\nu$ satisfying (13b) and the embedding of $(\mathcal{M},d)$ into $(\mathbb{R}^{d},\left\lVert\cdot\right\rVert)$ is a Lipschitz function.

5 Main result

In this section we state and discuss our main results, characterizing the expected distortion of reconstruction trees. The proofs are deferred to Section 6. Our first result is a probabilistic bound for any given threshold $\eta$. Recall that $s>0$ is defined by  $\eqref{eq:1}$ and $\hat{P}_{\eta}$ by (7) and (8).

Theorem 7**.**

Fix $\gamma>0$ as in $\eqref{eq:57}$ and $\eta>0$, for any $0<\sigma<s$

[TABLE]

where $c_{a}=\frac{1}{128(a+1)}$ and $c_{\mathbb{T}}>0$ depends on the partion tree $\mathbb{T}$.

As shown in Remark 19, it is possible to set $\sigma=s$ up to an extra logarithmic factor.

Next, we show how it allows derive a choice for $\eta$ as a function of the number of examples, and a corresponding expected distortion bound.

Corollary 8**.**

Fix $\gamma>1$, $\beta>0$ and set

[TABLE]

where $c_{a}=\frac{1}{128(a+1)}$. Then for any $0<\sigma<s$

[TABLE]

If $t$ is chosen large enough so that $\overline{c}_{\mathbb{T}}t-1=\beta$, then bound (16a) reads as

[TABLE]

where $c_{1}$ and $c_{2}$ are suitable constants depending on $\mathbb{T}$. This bound can be compared with Theorem 8 in [14] under the assumption that $\mathcal{M}$ is a compact $C^{\infty}$ manifold. Eq. (16a) with $s=1/d$ gives a convergence rate of the order $(\ln n/n)^{\frac{2p}{2p+d}}$ for any $p=\sigma/s\in(0,1)$, whereas the GMRA algorithm has a rate of the order $(\ln n/n)^{\frac{2}{2+d}}$, see also Proposition 3 of [14]. Hence, up to a logarithmic factor our estimator has the same convergence rate of the GMRA algorithm. However it is in order to notice that our algorithm works with a cheaper representation; indeed, given the adaptive partition $\hat{\Lambda}_{\eta}$, it only requires to compute and store the centers of mass $\{\hat{c_{I}}\}_{I\in\hat{\Lambda}_{\eta}}$.

In a similar setting, in [6], it is shown that the $k$-means algorithm with a suitable choice of $k=k_{n}$ depending on $n$, provides has a convergence rate of the order $(1/n)^{\frac{1}{d+1}}$ and $k$-flat algorithm of the order $(1/n)^{\frac{2}{d+4}}$.

The proof of Theorem 7 relies on splitting the error in several terms. In particular, it requires studying the stability to random sampling and the approximation properties of reconstruction trees. This latter result is relevant in the context of quantization of probability measures, hence of interest in its own right. We present this result first.

Towards this end, we need to introduce the infinite sample version of the reconstruction tree. For any cell $I\in\mathbb{T}$, denote the volume of the cell by

[TABLE]

the center of mass of the cell by

[TABLE]

where $x^{*}_{I}$ is an arbitrary point in $\mathcal{X}$. The local expected distortion in a cell by

[TABLE]

and

[TABLE]

Given the threshold $\eta>0$, define the subtree

[TABLE]

and let ${\Lambda}_{\eta}=\Lambda({\mathcal{T}}_{\eta})$ be the corresponding outerleaves. Lemma 14 shows that ${\mathcal{T}}_{\eta}$ is finite, so that by Lemma 11 ${\Lambda}_{\eta}$ is a partition and the corresponding nonlinear projection is

[TABLE]

so that the code vectors are the centers of mass of the cells.

Comparing the definition of $\mathcal{T}_{\eta}$ and $\hat{\mathcal{T}}_{\eta}$, we observe that $\hat{\mathcal{T}}_{\eta}$ is truncated at the depth $j_{n}$ given by (6), whereas $\mathcal{T}_{\eta}$ is not truncated, but its maximal depth is bounded by Lemma 17.

Given the above definitions, we have the following result.

Proposition 9**.**

Given $\eta>0$, for all $0<\sigma<s$

[TABLE]

Note that the bound is meaningful only if $0<\eta<1$. Indeed for $\eta\geq 1$ $\Lambda_{\eta}=\{\mathcal{X}\}$ and $\mathcal{E}(P_{\Lambda_{\eta}})\leq 1$, see Remark 15.

5.1 Approximation Error

The quantity $\mathcal{E}(P_{\Lambda_{\eta}})$ is called approximation error, by analogy with the corresponding definition in statistical learning theory, and it plays a special role in our analysis.

The problem of approximating a probability measure with a cloud of points is related to the so called optimal quantization [11]. The cost of an optimal quantizer is defined as:

[TABLE]

where $d(x,S)=\min_{y\in S}\|x-y\|$. An optimal quantizer corresponds to a set $S$ of $N$ points attaining the infimum, with the corresponding Voronoi-Dirichlet partition of $\mathcal{X}$. One can interpret the approximation error $\mathcal{E}(P_{\Lambda_{\eta}})$ as the quantization cost associated with the (suboptimal) quantizer given by the partition ${\Lambda}_{\eta}$ as defined in 17 with the corresponding centers $\{c_{I}\}_{I\in\Lambda_{\eta}}$, and $N:=\#{\Lambda}_{\eta}$.

This point of view is also taken through the analysis of $k$-means given in [6], optimal quantizers corresponding in fact to absolute minimizers of the $k$-means problem. Asymptotic estimates for the optimal quantization cost are available, see [6] and references therein. In the case of $\operatorname{supp}(\rho)=\mathcal{M}$, being $\mathcal{M}$ a smooth $d$-dimensional manifold isometrically emdedded in $\mathcal{X}$, they read:

[TABLE]

where $C(\rho)$ is a constant depending only on $\rho$. We underline that the result provided by Proposition 9 is actually a non-asymptotic estimate for the quantization cost, when the quantizer is given by the outcome of our algorithm. The quantization cost is strictly higher than the optimal one, since, for instance, an optimal quantizer always corresponds to a Voronoi-Dirichlet partition [11]. Nevertheless, as observed in Section 3.3, a Voronoi quantizer is not suitable for multiscale refinements, whereas ours is. Proposition 9 does not directly compare with (20), as it depends on a different parameter quantifying the complexity of the partition, namely $\eta$ instead of $N$. Though, by carefully applying (39b), in the manifold case we get:

[TABLE]

so that the bound is in fact optimal up to a logaritmic factor. Furthermore, it is in order to observe that Assumption 5 together with Proposition 9 provide a more transparent understanding of the approximation part of the analysis, as compared to what is provided in [4] and [14]. Therein, the approximation error is essentially addressed by defining the class of probability measures $\mathcal{B}_{s}$ as those for which a certain approximation property holds; see Definition 5 in [14] and Definition 5 in [4], in both being the thresholding algorithm explicitly used. On the other hand Assumption 5 does not depend on the thresholding algorithm, but only on the mutual regularity of $\rho$ and $\mathbb{T}$. Lastly we notice that, while for the sake of clarity none of the constants appear explicitly in our results, the proofs allow in principle to estimate them.

6 Proofs

In this section we collect some of the proofs of the above results. The more technical proofs are postponed to the appendix.

Proof of Thm. 6.

We first observe that it is enough to show that there exists a partition tree $\mathbb{T}^{\prime}=\{\Lambda_{j}\}$ for $\mathcal{M}$. Indeed, by adding to each partion $\Lambda_{j}$, the cell $I_{0}=\mathcal{X}\setminus\mathcal{M}$, we get a partion of $\mathcal{X}$, which satisfies Assumptions 5, since $\operatorname{diam}_{\rho}(I_{0})=0$.

Since $\mathcal{X}$ is bounded, then $\mathcal{M}$ is a connected compact manifold and, hence, $(\mathcal{M},d_{\mathcal{M}},\rho_{\mathcal{M}})$ is an Ahlfors regular metric measure space [10, p. 413], * i.e.*

[TABLE]

where $B_{\mathcal{M}}(x,r)$ is the ball of center $x$ and radius $r$ with respect to the Riemannian metric $d_{\mathcal{M}}$. By (13b)

[TABLE]

where $d$ is the intrinsic dimension of $\mathcal{M}$. Since $(\mathcal{M},d_{\mathcal{M}},\rho)$ is an Ahlfors regular metric measure, too, there exists a family of dyadic cubes, * i.e* for each $j\in\mathbb{Z}$ there is a family $\Lambda_{j}=\{I\}$ of open subsets of $\mathcal{M}$ such that

[TABLE]

where $0<r_{0}<r_{1}$ and $\delta\in(0,1)$ are given constants [7, Thm. 11]. As noted in [9], it is always possible to redefine each cell $I\in\Lambda_{j}$ by adding a suitable portion of its boundary in such a way that

[TABLE]

and (13a)–(22e) still hold true, possibly with different constants. Since $\mathcal{M}$ is compact, there exists $j_{0}\in\mathbb{Z}$ such that $B_{\mathcal{M}}(x_{0},r_{1}\delta^{j_{0}})=\mathcal{M}$ for some $x_{0}\in\mathcal{M}$. Hence, possibly redefining $j$, $r_{0}$ and $r_{1}$, we can assume that $\Lambda_{0}=\{\mathcal{M}\}$ and, as a consequence of (22b), (22c) and (23), the family $\{\Lambda_{j}\}_{j\in\mathbb{N}}$ is a partion tree for $\mathcal{M}$ where the bound in (3) is a consequence of the following standard volume argument. Fix $j_{0}$ large enough such that for all $j\geq j_{0}$, $r_{1}\delta^{j}\leq\operatorname{diam}{\mathcal{M}}$, then given $I\in\Lambda_{j}$

[TABLE]

where the third and the forth inequalities are consequence of (22d) and (21).

On the other hand, by (22e) and (21),

[TABLE]

so that

[TABLE]

Bound (3) holds true by setting

[TABLE]

We now show that (10b) holds true. Indeed, since $\mathcal{M}$ is Riemmannian submanifold of $\mathbb{R}^{D}$ it holds that

[TABLE]

see [22, Cor. 2, Prop. 21, Chapter 5]. Given $I\in\Lambda_{j}$, by (22e),

[TABLE]

so that (10b) holds true with $b=1/\delta>1$ and $C_{2}=2r_{1}$. To show (10a), given $I\in\Lambda_{j}$, by (22d) and (21)

[TABLE]

Hence

[TABLE]

so that (10a) holds true with $C_{1}=2r_{1}(d_{1}p_{1})^{-\frac{1}{d}}$ and $s=1/d$. ∎

The proof of Theorem 7 borrows ideas from [4, 14] and combines a number of intermediate results given in Appendix A. For sake of clarity let $\hat{P}_{\eta}=\widehat{P}_{\widehat{\Lambda}_{\eta}}$.

Proof of Thm. 7.

Consider the following decomposition

[TABLE]

which holds for all $x\in\mathcal{X}$. Since

[TABLE]

it holds that

[TABLE]

We bound the four terms.

1. A)

Since $\hat{\mathcal{T}}_{\eta}\cup\mathcal{T}_{2\eta}\supset\mathcal{T}_{2\eta}$, $\Lambda(\hat{\mathcal{T}}_{\eta}\cup\mathcal{T}_{2\eta})$ is a partition finer than $\Lambda_{2\eta}$, then

[TABLE]

where the last inequality is a consequence of (19). 2. B)

Bound (50a) implies that the term $B$ is zero with probability greater than $1-p_{B}$, where

[TABLE] 3. C)

Since $\Lambda(\hat{\mathcal{T}}_{\eta}\cap\mathcal{T}_{\eta/2})\subset\mathcal{T}_{\eta/2}\cup\Lambda_{\eta/2}=\mathcal{I}$ and $\sharp\Lambda(\hat{\mathcal{T}}_{\eta}\cap\mathcal{T}_{\eta/2})\leq\sharp\Lambda_{\eta/2}=N$, by (41) term C is bounded by $t^{*}=\eta^{\frac{4\sigma}{2\sigma+1}}t$ with probability greater than $1-p_{C}$ with

[TABLE]

where the second inequality is a consequence of (39a) and (39b), and $c_{\mathbb{T}}>0$ is a suitable constant depending on the partition tree $\mathbb{T}$. 4. D)

By (50b) term D is zero with probability greater that $1-p_{D}$ where

[TABLE]

If follows that with probability greater than $1-(p_{B}+p_{C}+p_{B})$

[TABLE]

i.e.

[TABLE]

which gives (14). ∎

Proof of Cor. 8.

Since $\eta_{n}^{2}=\frac{(\gamma+\beta)\ln n}{c_{a}n}$, then bound (14) gives (16a) since

[TABLE]

where $\overline{c}_{\mathbb{T}}=c_{\mathbb{T}}c_{a}^{-1}(\gamma+\beta)$. Eq. (16b) is clear. ∎

Proof of Prop. 9.

Given $I\in\mathbb{T}$, by (34) and (10b),

[TABLE]

so that

[TABLE]

and, by taking the limit in (32),

[TABLE]

Set $\mathcal{T}_{k}=\mathcal{T}_{\eta/2^{k}}$ for all $k\in\mathbb{N}$, then

[TABLE]

where the first inequality is a consequence of the fact that $\epsilon_{J}<(\frac{\eta}{2^{k}})^{2}$ if $J\notin\mathcal{T}_{k}$, the second inequality follows from (39a), whereas the last inequality holds since the series $\sum_{k=0}^{+\infty}4^{-\frac{2\sigma k-1}{2\sigma+1}}$ converges. ∎

7 Conclusions

In this paper, we proposed and analyzed a multiscale vector quantization approach inspired by ideas in [14]. We provided non asymptotic error bounds on the corresponding distortion/reconstruction error combining geometric and probabilistic tools. The analysis is developed in a general setting with manifold supported data as a special case.

A number of research directions remain to be explored. Following again [14], it would be interesting to understand the role of noise and the impact of data driven partitioning. Further, following the parallel with wavelets it would be interesting to use results from reproducing kernel Hilbert spaces to develop Gabor like geometric wavelets and analyse the properties of the corresponding quantization schemes.

Acknowledgments

L. R. acknowledges the financial support of the AFOSR projects FA9550-17-1-0390 and BAA-AFRL-AFOSR-2016-0007 (European Office of Aerospace Research and Development), and the EU H2020-MSCA-RISE project NoMADS - DLV-777826.

Appendix A Further Proofs

This section is devoted to show all the results cited in the proof in Section 6. We first show some preliminary results.

A.1 Technical results

We introduce some further notations about partition trees.

With slight abuse of notation, we regard ${\mathbb{T}=\cup_{j\in\mathbb{N}}\cup_{I\in\Lambda_{j}}I}$ as the (disjoint) union of the cells in each partition $\Lambda_{j}$. It a cell does not split, * i.e.* $\mathcal{C}(I)=I$, we regard $I\in\Lambda_{j}$ and $\mathcal{C}(I)\in\Lambda_{j+1}$ as different cells.

Given a cell $I\in\mathbb{T}$, for any $k\in\mathbb{N}$ we set

[TABLE]

where $\mathcal{C}^{0}(I)=\mathcal{P}^{0}(I)=\{I\}$ and, clearly,

[TABLE]

Furthermore, for any pair $I,J\in\mathbb{T}$, $I\neq J$ one and only one of the following alternative possibilities holds true

[TABLE]

for some $k\geq 1$.

If $\{\mathcal{T}_{t}\}_{t\in T}$ is an arbitrary family of subtrees, clearly the intersection $\cap_{t\in T}\mathcal{T}_{t}$ and the union $\cup_{t}\mathcal{T}_{t}$ are the * smallest * and the * largest* subtrees in the family.

Given a subset $S\subset\mathbb{T}$, we set

[TABLE]

which is the smallest subtree containing all the cells in $S$, and

[TABLE]

The following remark provides an alternative definition of $\hat{\mathcal{T}}_{\eta}$ and a similar procedure can be applied to $\mathcal{T}_{\eta}$

Remark 10**.**

Given $\eta>0$ and $j_{n}\in\mathbb{N}$, set

[TABLE]

which is a finite subset of $\mathbb{T}$. It is easy to check that $\mathcal{T}(\hat{S}_{\eta})=\hat{T}_{\eta}$.

The following lemma is quite obvious.

Lemma 11**.**

If $\mathcal{T}$ is finite subtree, the family of outer leaves

[TABLE]

is a partition with

[TABLE]

Proof.

For any $j\in\mathbb{N}$, we fix an arbitrary order among the cells in $\Lambda_{j}$, * i.e.*

[TABLE]

Given two different cells $I=I_{j,k}$ and $J=I_{j^{\prime},k^{\prime}}$ in $\mathbb{T}$, we say that $I$ is older than $J$ either if $j<j^{\prime}$ or $j=j^{\prime}$ and $k<k^{\prime}$.

By definition $\mathcal{T}$ contains the parents of all its elements and, hence, the root $\mathcal{X}$. Define, by induction, the family of subtrees

[TABLE]

where $I$ is the oldest cell in $\mathcal{T}\setminus\mathcal{T}_{n}$. Note that, by construction, $I\in\Lambda(\mathcal{T}_{n})$. Since $\mathcal{T}$ is finite by assumption, $\mathcal{T}_{N}=\mathcal{T}$ with $N=\sharp\mathcal{T}$.

We now prove by induction on $n=1,\ldots,N$ that $\Lambda(\mathcal{T}_{n})$ is a partition. If $n=1$

[TABLE]

which is a partition by assumption and, by (3), it satisfies (28). Assume that $\Lambda(\mathcal{T}_{n})$ is partition satisfying (28) with $\mathcal{T}=\mathcal{T}_{n}$. By construction, $\mathcal{T}_{n+1}=\mathcal{T}_{n}\cup I$ with $I\in\Lambda(\mathcal{T}_{n})$, then

[TABLE]

which is a partition since $I=\cup_{J\in C(I)}J$ and

[TABLE]

so that (28) holds true with $\mathcal{T}=\mathcal{T}_{n+1}$. ∎

We observe that, given a cell $I$, by definition of $\operatorname{diam}_{\rho}(I)$, it exists a measurable subset $J\subset I$ such that $\rho(J)=0$ and $\operatorname{diam}(I\setminus J)=\operatorname{diam}_{\rho}(I)$. Furthermore,

[TABLE]

The following simple lemmas will be useful.

Lemma 12**.**

Given a cell $I\in\mathbb{T}$ with $\rho(I)>0$

[TABLE]

Proof.

By definition of essential diameter, there exists $I_{0}\subset I$ such that $\operatorname{diam}(I_{0})=\operatorname{diam}_{\rho}(I)$ and $\rho(I\setminus I_{0})=0$. Let $C$ the closed convex hull of $I_{0}$. It is known that $\operatorname{diam}(C)=\operatorname{diam}_{\rho}(I)$ and, by convexity theorem, see [27, Thm. 5.7.35],

[TABLE]

so that (29a) is clear. Since $J,K\subset I$, Eq. (29b) is a consequence of the fact that $c_{J},c_{K}\in C$. If $n_{I}=0$, $\widehat{c}_{I}=\hat{x}_{I}^{*}\in\mathcal{X}$ so that (29c) is clear. If $n_{I}\neq 0$, almost surely $\widehat{c}_{I}\in C$ so that

[TABLE]

∎

Given a cell $I\in\mathbb{T}$, the within-between decomposition of the variance

[TABLE]

implies

[TABLE]

As a consequence we have the following decomposition.

Lemma 13**.**

Given $I\in\mathbb{T}$, for all $N\in\mathbb{N}$

[TABLE]

Proof.

The claim is clear for $N=0$. Assume that it holds true for $N$. Then, for any $J\in\mathcal{C}^{N+1}(I)$

[TABLE]

hence

[TABLE]

by observing that a cell $J^{\prime}\in\mathcal{C}(J)$ for some $J\in\mathcal{C}^{N+1}(I)$ if and only if $J^{\prime}\in\mathcal{C}^{N+2}(I)$, so that (32) holds true for $N+1$. ∎

We now show that the set $\mathcal{T}_{\eta}$ defined by (17) is a finite subtree.

Lemma 14**.**

The family $\mathcal{T}_{\eta}$ is a finite subtree of $\mathbb{T}$.

Proof.

If $\mathcal{T}_{\eta}=\{\mathcal{X}\}$, there is nothing to prove. Otherwise, if $I\in\mathcal{T}_{\eta}$, then by definition there exists $J\in\mathbb{T}$ such that $J\subset I$ and $\epsilon_{J}\geq\eta$. Since $\mathcal{P}(I)\supset I\supset J$, then $P(I)\in\mathcal{T}_{\eta}$, so that $\mathcal{T}_{\eta}$ is a subtree.

We now show that $\mathcal{T}_{\eta}$ is finite. From (32) with $I=\mathcal{X}$ and (34) we get that for all $N\in\mathbb{N}$

[TABLE]

Then the series $\sum_{k=0}^{+\infty}\sum_{J\in\mathcal{C}^{k}(\mathcal{X})}\epsilon_{J}^{2}=\sum_{I\in\mathbb{T}}\epsilon_{J}^{2}$ is sommable. Hence, the set

[TABLE]

is finite. Furthermore, by construction

[TABLE]

Bound (27b) implies that $\mathcal{T}_{\eta}$ is finite. ∎

Remark 15**.**

By (9),

[TABLE]

and $\epsilon_{I}<\mathcal{E}_{I}\leq 1$ provided that $\epsilon_{I}>0$. Furthermore, by (34) $\mathcal{T}_{\eta}=\Lambda_{\eta}=\{\mathcal{X}\}$ for all $\eta\geq 1$ and $\mathcal{E}(P_{\Lambda_{\eta}})=\mathcal{E}_{\mathcal{X}}\leq 1$. By the same argument $\hat{\mathcal{T}}_{\eta}=\hat{\Lambda}_{\eta}=\{\mathcal{X}\}$. Hence, it is enough to consider the case $0<\eta<1$.

A.2 $A$-term: Approximation error

In this section, we bound the approximation error, which is based in an estimation of the number of cells $I$ such that $\epsilon_{I}$ is big enough.

Lemma 16**.**

Given a partition $\Lambda\subset\mathbb{T}$, given $0<\eta<1$,

[TABLE]

Proof.

First inequality in (35) is a direct consequence of (34). By (10a) there exists $C>0$ such that for all $I\in\mathbb{T}$

[TABLE]

then, by (34)

[TABLE]

Set $\Lambda_{+}=\{I\in\Lambda:{\mathcal{E}}_{I}\geq\eta^{2}\}$ and $N_{+}=\sharp\Lambda_{+}$. Fix $q\geq 1$, clearly

[TABLE]

The Holder inequality with $1/p+1/q=1$ gives

[TABLE]

where the last inequality follows by choosing $\frac{p(2s+1)}{q}=1$,

• i.e.* $q=2s+2$. By replacing in (36)

[TABLE]

and, since $1-\frac{1}{2s+2}=\frac{2s+1}{2s+2}$, we get (35). ∎

We first observe that the proof of the above lemma only depends on Assumption (10a) and the constant in the inequality (35) only depends on the constant in (10a), denoted by $C$ in the proof. Furthermore, without Assumption (10a) we always have the following bound

[TABLE]

where the first inequality is consequence of (34) and the last bounds is due to (30) with $I=\mathcal{X}$

[TABLE]

We recall that $\Lambda_{\eta}=\Lambda(T_{\eta})$ is the family of the corresponding outer leaves, which is a partition of $\mathcal{X}$ by Lemma 11. In order to bound the cardinality of $\Lambda_{\eta}$ we need an auxiliary lemma based on Assumption (10b).

Lemma 17**.**

Given $\eta>0$, set

[TABLE]

then

[TABLE]

If $\{j_{I}\in\mathbb{N}\mid I\in\mathbb{T}\text{ and }\epsilon_{I}\geq\eta\}=\emptyset$ we set $j_{\eta}=0$.

Proof.

If $\mathcal{T}_{\eta}=\{\mathcal{X}\}$ or $\{j_{I}\in\mathbb{N}\mid I\in\mathbb{T}\text{ and }\epsilon_{I}\geq\eta\}=\emptyset$, then $j_{\eta}=0$, so that the claim is evident. If $\mathcal{T}_{\eta}\neq\{\mathcal{X}\}$, then $0<\eta<1$. Take $I\in\mathbb{T}$ such that $\epsilon_{I}\geq\eta$. By (34) and (10b)

[TABLE]

Hence

[TABLE]

where $E=\max\{1,\frac{\ln C_{2}}{2\ln 2}\}/\ln b$. ∎

Proposition 18**.**

Given $\eta>0$, for all $\sigma<s$,

[TABLE]

where the constants in $\lesssim$ also depend on $\sigma$.

Proof.

As observed in Remark 15, we can assume that $0<\eta<1$. Let

[TABLE]

By (26) the elements of $\Upsilon$ are disjoint. Let $\Lambda\subset\mathbb{T}$ be a partition such that $\Upsilon\subset\Lambda$. Hence, by (35)

[TABLE]

We claim that

[TABLE]

By construction $\mathcal{T}_{\eta}\supset\bigcup_{J\in\Upsilon}\mathcal{\mathcal{T}}(J)$. To prove the opposite inclusion, fix $I\in\mathcal{T}_{\eta}$, then there exists $J_{1}\in\mathbb{T}$ such that $J_{1}\subset I$ and $\epsilon_{J_{1}}\geq\eta$. If $J_{1}\in\Upsilon$, then $I\in\mathcal{T}(J_{1})$. Otherwise, there exists $J_{2}\subset J_{1}\subset I$ and $\epsilon_{J_{2}}\geq\eta$. If $J_{2}\in\Upsilon$, $I\in\mathcal{T}(J_{2})$. Otherwise, because of $\mathcal{T}_{\eta}$ is finite, we can repeat the procedure until we get $J_{k}\in\Upsilon$ such that $J_{k}\subset I$, then $I\in\mathcal{T}(J_{k})$ and the claim is proven. By (27b)

[TABLE]

since, by definition, $j_{J}\leq j_{\eta}$ and the last inequality is due to (40) and (38). This shows the first inequality in (39a). Since $\sigma<s$, for some $\delta>0$

[TABLE]

where $C=\sup_{0<\eta\leq 1}\eta^{\delta}\ln(\frac{2}{\eta})$, which is finite, since $\lim_{\eta\to 0}\eta^{\delta}\ln(\frac{2}{\eta})=0$. Bound (39b) is a direct consequence of (28). ∎

Remark 19**.**

In the following, for sake of clarity we bound the logarithmic dependence on $\eta$ by considering $\sigma<s$. However our results can be extended to $\sigma=s$ by adding a logarithmic factor, as in (39a) and (39b).

A.3 $C$-term: sample error

The following result bounds the sample error for a given partition.

Proposition 20**.**

Fix a data-independent subset $\mathcal{I}\subset\mathbb{T}$ of cells and $N>0$. Given a partition $\widehat{\Lambda}\subset\mathcal{I}$ (possibly depending on the data) such that $\sharp\widehat{\Lambda}\leq N$, for any $t>0$,

[TABLE]

Proof.

Consider the following event

[TABLE]

which is well-defined since $\mathcal{I}$ does not depend on the data $X_{1},\ldots,X_{n}$. By union bound

[TABLE]

Fix $I\in\mathcal{I}$ with $\rho_{I}>0$. The tower property with respect to the binomial random variable $n_{I}$ gives

[TABLE]

Conditionally to the event $\{n_{I}=k\}$ with $k>0$, up to a permutation of the indexes, we can assume that $X_{1},\ldots,X_{k}\in I$ and $X_{k+1},\ldots,X_{n}\notin I$. Furthermore,

[TABLE]

where $\xi_{1},\ldots,\xi_{k}$ are independent zero mean random vectors bounded by $M=\operatorname{diam}_{\rho}(I)$ almost surely by (29a). Hence, by (56)

[TABLE]

which trivially holds true also if $k=0$. Hence,

[TABLE]

where in the fourth line we use the bound $(1-\tau)^{n}\leq\exp(-n\tau)$ with $0\leq\tau\leq 1$. Since

[TABLE]

then, for all $t\leq\operatorname{diam}_{\rho}(I)$,

[TABLE]

If $\operatorname{diam}_{\rho}(I)<t\leq\operatorname{diam}{\mathcal{X}}\leq 1$, by (29c), clearly $\mathds{P}[\left\lVert\widehat{c}_{I}-c_{I}\right\rVert>t\mid n_{I}>0]=0$, so that

[TABLE]

where the last bound holds true for any $t\leq 2\sqrt{2}$. Finally, if $t>\operatorname{diam}{\mathcal{X}}$, as above

[TABLE]

since $\widehat{x}^{*}_{I},c_{I}\in\mathcal{X}$, compare with (29c). It follows that (43) holds true for all $t>0$. Summarizing, from (42) we get

[TABLE]

Since $\widehat{\Lambda}\subset\mathcal{I}$, on the complement of $\Omega$,

[TABLE]

and bound (41) follows by replacing $t$ with $\sqrt{t/N}$. ∎

Remark 21**.**

By inspecting the proof, it is possible to check that the assumption $\hat{x_{I}}^{*}\in\mathcal{X}$ is needed only in this proposition and it can be replaced by the condition that $\inf_{x\in\mathcal{X}}\left\lVert\hat{x}_{I}^{*}-x\right\rVert\leq 1$, so that $\left\lVert\hat{x}_{I}^{*}-c_{I}\right\rVert\leq 2$.

A.4 $B$ and $D$ terms: Stability of $P_{\Lambda}$ with respect to

the partition.

The following result is well-known.

Lemma 22**.**

Given a cell $I\in\mathbb{T}$ with $\rho_{I}>0$, for all $t>0$

[TABLE]

where $M_{I}=2/3\max\{4,(1+2\rho_{I})/\rho_{1}\}$.

Proof.

We apply the Bernstein inequality, see [5, Cor. 2.11], to the family of independent random variables $\mathbbm{1}_{I}(X_{1}),\ldots,\mathbbm{1}_{I}(X_{n})$, which satisfy

[TABLE]

then,

[TABLE]

Observing that

[TABLE]

if $t>\max\{1,1/\rho_{I}-1\}=t^{*}$, then $\mathds{P}\left[\left|n_{I}-n\rho_{I}\right|\geq n\rho_{I}t\right]=0$, whereas if $t\leq\max\{1,1/\rho_{I}-1\}=t^{*}$, it holds that

[TABLE]

so that the second bound in (44) is clear. ∎

The following lemma provides a concentration inequality of $\sqrt{\frac{n_{I}}{n}}$.

Lemma 23**.**

Given a cell $I\in\mathbb{T}$ with $\rho_{I}>0$, for all $t>0$

[TABLE]

Proof.

We apply Proposition 27 with $\mathcal{Y}=\{0,1\}$

[TABLE]

where $f$ is clearly bounded, and

[TABLE]

Given $k=1,\ldots,n$, it holds that

[TABLE]

so that $\alpha=1/n$. Furthermore

[TABLE]

then $\beta=1/n$ and Eq. (58) implies (45). ∎

The following lemma shows that, given a cell $I\in\mathbb{T}$, $\widehat{\epsilon}_{I}$ concentrates around $\epsilon_{I}$.

Lemma 24**.**

Given $I\in\mathbb{T}$, for all $t>0$

[TABLE]

where $\ell=1+\sharp\mathcal{C}(I)$.

Proof.

Fix $I\in\mathbb{T}$. If $\rho_{J}=0$ for some $J\in\mathcal{C}(I)$, then almost surely $X_{i}\notin J$ for all $i=1,\ldots,n$ and, hence, $n_{J}=0$, so that both $\widehat{\epsilon}_{I}$ and $\epsilon_{I}$ do not depend on the children $J$. Hence, without loss of generality, we can assume that $\rho_{J}>0$ for all $J\in\mathcal{C}(I)$.

Let $\ell=\sharp\mathcal{C}(I)+1$. Set $L^{2}(\mathcal{C}(I))=\mathbb{R}^{\ell-1}$, regarded as Euclidean vector space whose norm is denoted by $\left\lVert\cdot\right\rVert_{2}$. Define $v,\widehat{v},\widehat{w}\in L^{2}(\mathcal{C}(I))$

[TABLE]

Then

[TABLE]

We now bound the first term. Set $\mathcal{I}=\mathcal{C}(I)+\{I\}$ and $\sharp\mathcal{I}=\ell$. Then

[TABLE]

Setting

[TABLE]

bound (43) gives

[TABLE]

so that

[TABLE]

We now bound the second term in (47). Set

[TABLE]

bound (44) gives

[TABLE]

On the complement on $\Omega_{I}^{\prime}$,

[TABLE]

Hence

[TABLE]

Inequality (47) with bounds (48) and (49) implies (46). ∎

The next proposition shows that $P_{\Lambda}$ is stable under suitable small perturbations of the partition $\Lambda$.

Proposition 25**.**

For any $\eta>0$

[TABLE]

Proof.

Recalling the definition of $j_{n}$ given by (6), set $\mathcal{T}^{*}_{n}=\bigcup\limits_{j\leq j_{n}}\Lambda_{j}$, which is a subtree with

[TABLE]

and, by construction, $\widehat{T_{\eta}}\subset\mathcal{T}^{*}_{n}$. The probability of the event in the left hand side of (50a) is clearly bounded by the probability of the event

[TABLE]

About the first term, we observe that if $I\in\hat{\mathcal{T}}_{\eta}\subset\mathcal{T}^{*}_{n}$, then there exist $k\geq 0$ and $J\in\mathcal{C}^{k}(I)\cap\mathcal{T}^{*}_{n}$ such that $\widehat{\epsilon}_{J}\geq\eta$ and, since $I\notin\mathcal{T}_{\eta/2}$ and $J\in\mathcal{C}^{k}(I)$, then $\epsilon_{J}<\frac{\eta}{2}$, so that

[TABLE]

By union bound and (46) with $t=\eta/2$, $\operatorname{diam}_{\rho}(I)\leq 1$ and $\ell\leq a+1$ give

[TABLE]

where the second inequality is a consequence of (51) and $c_{a}$ is given by (50c).

By a similar argument

[TABLE]

By union bound and (46) with $t=\eta$ and $\operatorname{diam}_{\rho}(I)\leq 1$ give

[TABLE]

where the second inequality is a consequence of (39a).

By (53) and (52), we get (50a). The proof of (50b) can be deduced reasoning as for (53). ∎

Appendix B Auxialiry results

We recall the following probabilistic inequality based on a result of [24, 25], see also [31, Thm. 3.3.4] and [23] for concentration inequalities for Hilbert-space-valued random variables.

Proposition 26**.**

Let $\xi_{1},\ldots,\xi_{n}$ be a family of independent zero-mean random variables taking values in a real separable Hilbert space and satisfying

[TABLE]

where $\Sigma$ and $M$ are two positive constants. Then, for all $n\in\mathbb{N}$ and $t>0$

[TABLE]

where $g(t)=\frac{t^{2}}{1+t+\sqrt{1+2t}}$. In particular, if $\xi_{i}$ are bounded by $M$ with probability 1, then

[TABLE]

Proof.

Bound(55) is given in [24] with a wrong factor, see [25]. To show(56), note that (54) is satisfied with $\Sigma=M$. Furthermore, for all $t\leq 1$, $g(t)\geq t^{2}/4$, so that if $t\leq M$,

[TABLE]

whereas, if $t>M$, (56) is trivially satisfied. ∎

The following concentration inequality is based on [18] and we adapt the proof of Theorem 10 in [19]. We introduce the following notation. Given a family $\xi_{1},\ldots\xi_{n}$ of independent random variables taking value in some measurable space $\mathcal{Y}$ and a measurable positive bounded function $f:\mathcal{Y}^{n}\to\mathbb{R}$, for any $k=1,\ldots,n$ set

[TABLE]

Proposition 27**.**

With the above notation, if there exist two constants $\alpha,\beta>0$ such that

[TABLE]

then, for any $t>0$

[TABLE]

Proof.

Set $Z_{k}=V_{k}/\alpha$ and $Z=f(\xi_{1},\ldots,\xi_{n})/\alpha$. By construction

[TABLE]

Let $\gamma=\max\{\beta/\alpha,1\}$. Theorem 13 of [18] gives that

[TABLE]

By replacing $t$ with $2t\sqrt{\mathbb{E}[Z]}$ in (59)

[TABLE]

since

[TABLE]

provided that $\left|\sqrt{\mathbb{E}[Z]}-t\right|\leq\sqrt{Z+t^{2}}$. Hence

[TABLE]

Setting $\frac{t^{2}}{2\mathbb{E}[Z]+t}=2\tau^{2}$, bound (60) gives

[TABLE]

so that, setting $\tau=t/2$,

[TABLE]

Bounds (61) and (62) imply that

[TABLE]

and, by replacing $t$ with $t/\sqrt{\alpha}$,

[TABLE]

where $\alpha\gamma=\alpha\max\{\beta/\alpha,1\}=\max\{\beta,\alpha\}$. ∎