New Risk Bounds for 2D Total Variation Denoising

TL;DR

This paper investigates the adaptivity of 2D Total Variation Denoising (TVD) for piecewise constant functions, showing it performs better than worst-case guarantees and proposing a data-driven tuning method.

Contribution

It provides new theoretical risk bounds for TVD when the true function is piecewise constant and introduces a data-driven tuning procedure with strong guarantees.

Findings

TVD performs better on piecewise constant functions than in worst-case scenarios.

A fully data-driven tuning method for TVD achieves near-optimal risk guarantees.

Theoretical analysis confirms adaptivity of TVD to certain structured functions.

Abstract

2D Total Variation Denoising (TVD) is a widely used technique for image denoising. It is also an important nonparametric regression method for estimating functions with heterogenous smoothness. Recent results have shown the TVD estimator to be nearly minimax rate optimal for the class of functions with bounded variation. In this paper, we complement these worst case guarantees by investigating the adaptivity of the TVD estimator to functions which are piecewise constant on axis aligned rectangles. We rigorously show that, when the truth is piecewise constant, the ideally tuned TVD estimator performs better than in the worst case. We also study the issue of choosing the tuning parameter. In particular, we propose a fully data driven version of the TVD estimator which enjoys similar worst case risk guarantees as the ideally tuned TVD estimator.

Figures

Equations

$$\text{mbox}all((i,j),(k,\ell ))\in L_n\times L_n\text{mbox}suchthat|i-j|+|k-\ell |=1.$$

$${\mathrm{TV}}_{\mathrm{norm}}(\theta ):=\frac{1}{n}\sum _{(u,v)\in E_n}|{\theta }_u-{\theta }_v|=\frac{1}{n}\Vert D\theta {\Vert }_1$$

$${\hat{\theta }}_{\text{V}}:={\mathrm{argmin}}_{\theta \in {\mathbb{R}}^{n\times n}:{\mathrm{TV}}_{\mathrm{norm}}(\theta )\le \text{V}}\Vert y-\theta {\Vert }^2$$

$$\mathrm{MSE}({\hat{\theta }}_{\text{V}},{\theta }^{\ast }):={\mathbb{E}}_{{\theta }^{\ast }}\frac{\Vert {\hat{\theta }}_{\text{V}}-{\theta }^{\ast }{\Vert }^2}{N}$$

$${\hat{\theta }}_{\lambda }:={\mathrm{argmin}}_{\theta \in {\mathbb{R}}^{n\times n}}\Vert y-\theta {\Vert }^2+\lambda {\mathrm{TV}}_{\mathrm{norm}}(\theta )$$

$$\mathrm{TV}(v):=\sum _{i=1}^{n-1}|v_{i+1}-v_i|.$$

$$\mathrm{MSE}({\hat{\theta }}_{\lambda },{\theta }^{\ast })\le C(\log n{)}^2\min \{\sigma \frac{{\mathrm{TV}}_{\mathrm{norm}}({\theta }^{\ast })}{\sqrt{N}},{\sigma }^2\frac{\Vert D{\theta }^{\ast }{\Vert }_0}{N}\}.$$

$$\theta^{*}=\left[{\begin{array}[]{cc}\mathbf{0}_{n\times n/2}&\mathbf{1}_{n\times n/2}\\ \end{array}}\right]$$

$$\mathrm{TV}(\theta ):=\sum _{(u,v)\in E_n}|{\theta }_u-{\theta }_v|.$$

$${\rm MSE}(\widehat{\theta}_{\mathbf{V}},\theta^{*})\leq C\big{(}\sigma\frac{\mathbf{V}}{\sqrt{N}}\,(\log\mathrm{e}N)^{5/2}+\frac{\sigma^{2}}{N}\big{)}.$$

$${\rm MSE}(\widehat{\theta}_{\mathbf{V}},\theta^{*})\leq C\big{[}(\mathbf{V}-\mathbf{V}^{*})^{2}+\sigma^{2}(\log\mathrm{e}n)^{9}\frac{\>k(\theta^{*})^{5/4}}{N^{3/4}}\big{]}\,.$$

$$\theta^{*}=\left[{\begin{array}[]{cc}\mathbf{0}_{n\times n/2}&\mathbf{1}_{n\times n/2}\\ \end{array}}\right]$$

$$\underset{\sigma \to 0}{\lim }\frac{1}{{\sigma }^2}\mathrm{MSE}({\hat{\theta }}_{{\mathbf{V}}^{\ast }},{\theta }^{\ast })\ge \frac{c}{N^{3/4}}.$$

$$\mathcal{G}\mathcal{W}(K)=\mathbb{E}\underset{\theta \in K}{\sup }⟨Z,\theta ⟩$$

$$T_{K^{\ast }}({\theta }^{\ast }):=\mathrm{Closure}(\mathrm{Cone}(\{\theta \in {\mathbb{R}}^{n\times n}:{\theta }^{\ast }+\theta \in K^{\ast }\})).$$

$$\mathcal{G}\mathcal{W}(T_K(\theta )\cap B_{n,n}(1))\le C(\log n{)}^{4.5}k(\theta {)}^{5/8}{n}^{1/4}.$$

$$\theta^{*}=\left[{\begin{array}[]{cc}\mathbf{0}_{n\times n/2}&\mathbf{1}_{n\times n/2}\\ \end{array}}\right]$$

$$\mathcal{G}\mathcal{W}(T_{K^{\ast }}({\theta }^{\ast })\cap B_{n\times n}(1))\ge c{n}^{1/4}.$$

$$\mathcal{G}\mathcal{W}(T_{K^{\ast }}({\theta }^{\ast })\cap B_{n\times n}(1))\ge c{n}^{1/2}.$$

$$\underset{\sigma \to 0}{\lim }\frac{1}{{\sigma }^2}\mathrm{MSE}({\hat{\theta }}_{{\mathbf{V}}^{\ast }},{\theta }^{\ast })\ge \frac{c}{N^{1/2}}.$$

$${\hat{\theta }}_{\mathrm{notuning}}:={\mathrm{argmin}}_{\{v\in {\mathbb{R}}^{n\times n}:\overline{v}=0,\Vert y-\overline{y}1-v{\Vert }^2\le (n^2-1){\hat{\sigma }}^2\}}{\mathrm{TV}}_{\mathrm{norm}}(v)+\overline{y}1$$

$$\hat{\sigma }:=\frac{{\mathrm{TV}}_{\mathrm{norm}}(y)}{\mathbb{E}{\mathrm{TV}}_{\mathrm{norm}}(Z)}=\frac{\sqrt{\pi }{\mathrm{TV}}_{\mathrm{norm}}(y)}{4(n-1)}.$$

$${\rm MSE}(\widehat{\theta}_{\mathrm{notuning}},\theta^{*})\leq C\big{(}\sigma\frac{\mathbf{V}^{*}}{\sqrt{N}}\log(\mathrm{e}n)^{5/2}+\frac{(\mathbf{V}^{*})^{2}}{N}+\frac{\sigma^{2}}{\sqrt{N}}\big{)}\,$$

$$\theta^{*}=\left[{\begin{array}[]{cc}\mathbf{0}_{n\times n/2}&\mathbf{1}_{n\times n/2}\\ \end{array}}\right]$$

$$\underset{\sigma \downarrow 0}{\lim }\frac{1}{{\sigma }^2}\mathrm{MSE}({\hat{\theta }}_{{\mathbf{V}}^{\ast }},{\theta }^{\ast })<\underset{\sigma \downarrow 0}{\lim }\frac{1}{{\sigma }^2}\mathrm{MSE}({\hat{\theta }}_{\lambda },{\theta }^{\ast }).$$

$$\theta^{\mathrm{four}}\coloneqq\left[{\begin{array}[]{cc}1_{n/2\times n/2}&2_{n/2\times n/2}\\ 0_{n/2\times n/2}&1_{n/2\times n/2}\\ \end{array}}\right]$$

$$\mathcal{G}\mathcal{W}(K)=\mathbb{E}\underset{\theta \in K}{\sup }⟨Z,\theta ⟩$$

$$\sup_{\theta\in K}\>{\rm dist}\Big{(}\theta,\>\bigcup_{S\in\mathcal{S}}S\Big{)}\leq\epsilon$$

$$\mathcal{GW}(K)\leq C\sum_{k=k_{0}+1}^{k_{1}}2^{-k}\big{[}\max_{S\in\mathcal{S}_{2^{-k}}}\sqrt{2\>\dim(S)}+2\>\sqrt{\log|\mathcal{S}_{2^{-k}}|}+1\big{]}+\sqrt{n}\,2^{-k_{1}}$$

$$\mathrm{dist}(\theta ,{\theta }_k)=\mathrm{dist}(\theta ,{\mathcal{N}}_k).$$

Full text

and ††Author names are sorted alphabetically.

1 Introduction

Total variation denoising (TVD) is a standard technique to do noise removal in images. This technique was first proposed in rudin1992nonlinear and has since then been heavily used in the image processing community. It is well known that TVD gets rid of unwanted noise and also preserves edges in the image (see strong2003edge). For a survey of this technique from an image analysis point of view; see chambolle2010introduction and references therein.

The success of the TVD technique as a denoising mechanism motivates us to revisit this problem from a statistical perspective. In this paper, we are interested in the following statistical estimation problem. Consider observing $y=\theta^{*}+\sigma Z$ where $y\in\mathbb{R}^{n\times n}$ is a noisy matrix/image, $\theta^{*}$ is the true underlying matrix/image, $Z$ is a noise matrix consisting of independent standard Gaussian entries and $\sigma$ is an unknown standard deviation of the noise entries. Thus, in this setting, the image denoising problem is cast as a Gaussian mean estimation problem. Before defining the TVD estimator in this context, let us define total variation of an arbitrary matrix.

Let us denote the $n\times n$ two dimensional grid graph by $L_{n}$ and denote its edge set by $E_{n}$. More precisely, the vertices in $L_{n}$ correspond to the pairs $(i,j)\in[n]\times[n]$ and its edge set $E_{n}$ consists of:

[TABLE]

We will use $L_{n}$ interchangeably for the graph as well as the underlying set of vertices. Now, thinking of $\theta\in\mathbb{R}^{n\times n}$ as a function on $L_{n}$ let us define

[TABLE]

where $D$ is the usual edge vertex incidence matrix of size $2n(n-1)\times n^{2}.$ The $1/n$ factor is just a normalizing factor so that if $\theta_{ij}=f(i/n,j/n)$ for some underlying differentiable function on the unit square then $\mathrm{TV}_{\mathrm{norm}}(\theta)$ is precisely the discretized Riemann approximation for \int_{[0,1]^{2}}\big{|}\frac{\partial f(x,y)}{\partial x}\big{|}+\big{|}\frac{\partial f(x,y)}{\partial y}\big{|}. This $1/n$ scaling is termed as the canonical scaling in sadhanala2016total. The above notion of total variation extends the definition of variation from differentiable functions on the unit square to arbitrary matrices. We can now define the TVD estimator, which is our main object of study.

[TABLE]

where $\|.\|$ throughout this paper will denote the usual Frobenius norm for matrices. The TVD estimator is actually a family of estimators indexed by the tuning parameter $\textbf{V}>0.$ We will measure the performance of our estimator in terms of its normalized mean squared error (MSE) defined as

[TABLE]

where throughout this paper we denote $N=n^{2}.$

We defined the TVD estimator in its constrained form, however the penalized version is also popular in the literature, which is defined as follows:

[TABLE]

where $\lambda>0$ is a tuning parameter. In this paper, we focus on the analysis of the constrained version.

1.1 Background and Motivation

The $1$D version of this problem is a well studied problem (see, e.g. tibshirani2005sparsity) in nonparametric regression. In this setting, we again have $y=\theta^{*}+\sigma Z$ as before, where $y,\theta^{*},Z$ are now vectors instead of matrices. The total variation of a vector $v\in\mathbb{R}^{n}$ can now be defined as

[TABLE]

Again the above definition can be seen as a discrete Riemann approximation to \int_{[0,1]}\big{|}f^{{}^{\prime}}(x)\big{|}dx when $v_{i}=f(i/n)$ for some differentiable function $f.$ The constrained and the penalized versions of the TVD estimator can now be defined analogously. The penalized form seems to be more popular in the existing literature; in this case the TVD estimator is often referred to as fused lasso (see tibshirani2005sparsity, rinaldo2009properties). In this 1D setting, it is known (see, e.g. donoho1998minimax, mammen1997locally) that the TVD estimator is minimax rate optimal on the class of all bounded variation signals $\{\theta:\mathrm{TV}(\theta)\leq\mathbf{V}\}$ for $\mathbf{V}>0$. It is also shown in donoho1998minimax that no estimator, which is a linear function of $y$, can attain this minimax rate.

It is also worthwhile to mention here that TVD in the 1D setting has been studied as part of a general family of estimators which penalize discrete derivatives of different orders. These estimators have been studied in steidl2006splines, tibshirani2014adaptive and by kim2009ell_1 who coined the name trend filtering. A continuous version of these estimators, where discrete derivatives are replaced by continuous derivatives, was proposed much earlier in the statistics literature by mammen1997locally under the name locally adaptive regression splines.

Total variation of a signal can actually be defined over an arbitrary graph as the sum of absolute differences of the signal across edges of the graph. Trend Filtering on general graphs has been a popular research topic in the recent past; see wang2016trend, lin2016approximate. A more recent paper, ortelli2018, studies TVD on tree graphs. The 1D setting corresponds to the chain graph on $n$ vertices whereas the 2D setting corresponds to the 2D lattice graph on $n^{2}=N$ vertices.

The 2D TVD problem, while being much less studied than in its 1D counterpart, has enjoyed a recent surge of interest. Worst case performance of the TVD estimator has been studied in hutter2016optimal, sadhanala2016total, ortelli2019oracle. These results show that like in the 1D setting, the 2D TVD estimator is nearly minimax rate optimal over the class $\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}_{\mathrm{norm}}(\theta)\leq\textbf{V}\}$ of bounded variation signals. In fact, sadhanala2016total also generalize the result of donoho1998minimax and prove that no linear function of $y$ can attain the minimax rate in the 2D setting as well. A representative of the state of the art risk bound for the TVD estimator in 2D setting is due to hutter2016optimal (see also ortelli2019oracle). They studied the penalized form of the TVD estimator and proved that there exist universal constants $C,\>c>0$ such that by setting $\lambda=c\sigma\log n,$ one gets

Theorem 1.1** (Hutter Rigollet).**

[TABLE]

where $\|\cdot\|_{0}$ is the usual $\ell_{0}$ norm.

For convenience, we will henceforth use the usual $O(\cdot)$ notation to compare sequences. We write $a_{n}=O(b_{n})$ if there exists a constant $C>0$ such that $a_{n}\leq C\>b_{n}$ for all sufficiently large $n$. We also use $a_{n}=\widetilde{O}(b_{n})$ to denote $a_{n}=O(b_{n}(\log n)^{C})$ for some $C>0$.

In words, the bound in Theorem 1.1 is a minimum of two terms. The first term gives the $\ell_{1}$ rate scaling like $O(1/\sqrt{N})$ for bounded variation functions. The second one is the $\ell_{0}$ rate which can be much faster than the $O(1/\sqrt{N})$ rate if $|D\theta^{*}|_{0}$ is small enough. In spite of the above works, there are still a couple of unexplored aspects regarding 2D TVD, specifically its adaptivity to piecewise constant signals and minimax optimality without tuning, which are the focus of the present paper. We discuss them now.

1.1.1 Adaptivity to piecewise constant signals

Observe that the total variation semi norm is a convex relaxation for the number of times the true signal $\theta^{*}$ changes values along the neighbouring vertices. This fact suggests that the TV estimator might perform very well if the true signal is indeed piecewise constant. This phenomenon is now fairly well understood in the 1D setting. In this setting, suppose that the true vector $\theta^{*}$ is piecewise constant with $k+1$ contiguous pieces or blocks. Given data $y\sim N_{n}(\theta^{*},\sigma^{2}I_{n})$, an oracle estimator, which knows the locations of the jumps, would just estimate the signal $\theta^{*}$ by the mean of the data vector $y$ within each block. It can be easily checked that the oracle estimator will have MSE bounded by $\sigma^{2}(k+1)/n.$ Recent works (see dalalyan2017tvd, lin2016approximate) studied the penalized TVD estimator and showed that if the minimum length of the blocks where $\theta^{*}$ is constant is not too small (scales like $O(n/k)$) and if the tuning parameter $\lambda$ is set to be equal to an appropriate function of the unknown $\sigma$ and $n,$ then an oracle risk $O(k/n)$ could be achieved up to some additional logarithmic factors in $k$ and $n.$ In guntuboyina2017adaptive, this adaptive behaviour was established for the ideally tuned constrained form of the estimator with slightly better log factors. Thus, we can say that in the 1D setting, the TVD estimator is optimally adaptive to piecewise constant signals.

This motivates us to wonder whether similar adaptivity holds in the 2D setting. In this paper, we investigate adaptivity to signals/matrices which are piecewise constant on $k<<N$ axis aligned rectangles. Such adaptivity of the 2D TVD estimator has not been explored at all in the literature. Estimation of functions which are piecewise constant on axis aligned rectangles are naturally motivated by methodologies such as CART (see e.g breiman2017classification) which produce outputs of the same form. Recently, adaptation to piecewise constant structure on rectangles has been of interest in the nonparametric shape constrained function estimation literature also (see Theorem $2.3$ in chatterjee2018matrix and Theorems $2$ and $5$ in han2017isotonic). See Section 3.5 where we discuss some even more recent (which appeared after we uplaoded this paper) works about estimating piecewise constant functions on axis aligned rectangles. Here is the main question that we address in this paper.

Q1: If the underlying $\theta^{*}$ is piecewise constant on at most $k<<N$ axis aligned rectangles; can the ideally tuned TVD estimator attain a faster rate of convergence than the $\widetilde{O}(1/\sqrt{N})$ rate?

Basically we are asking the question whether the ideally tuned TVD estimator adapts to truths which are piecewise constant on a few axis aligned rectangles, which is a different notion of sparsity than the sparsity constraint of $\|D\theta^{*}\|_{0}$ being small. As a simple instance of $\theta^{*}$ being piecewise constant on rectangles, consider $\theta^{*}$ to be of the following form:

[TABLE]

In this case, we have $\|D\theta^{*}\|_{0}=O(\sqrt{N})$ and $\mathrm{TV}_{\mathrm{norm}}(\theta^{*})=O(1).$ Note that the $\ell_{0}$ bound of hutter2016optimal will give us an upper bound on the MSE scaling like $\widetilde{O}(1/\sqrt{N})$ which is already given by the $\ell_{1}$ bound. Thus, the result of hutter2016optimal does not help in answering our question and suggests there is no adaptation. In Theorem 2.2 of this paper, we show that the ideally tuned TVD estimator indeed adapts to piecewise constant matrices on axis aligned rectangles and provably attains a rate of convergence scaling like $\widetilde{O}(1/N^{3/4})$ which is strictly faster than the $\ell_{1}$ rate $\widetilde{O}(1/\sqrt{N})$. However, we also show that this $\widetilde{O}(1/N^{3/4})$ rate is tight and thus the TVD estimator is not able to attain the $\widetilde{O}(1/N)$ parametric rate that would be achieved by an oracle estimator. This is the main contribution of this paper and is the first result of its type in the literature as far as we are aware.

1.1.2 Minimax rate optimality without tuning

Existing results such as Theorem 1.1, along with minimax lower bounds shown in sadhanala2016total, show that the $\widetilde{O}(\tfrac{\textbf{V}}{\sqrt{N}})$ rate attained by the penalized TVD estimator is near minimax rate optimal. Thus we can say that the penalized TVD estimator is near minimax rate optimal over the parameter space $\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}_{\mathrm{norm}}(\theta)\leq\textbf{V}\}$, simultaneously over V and $N$. However, this penalized TVD estimator needs to set a tuning parameter $\lambda$ which depends on the unknown $\sigma$ and an implicit constant $C$ which can be potentially difficult to set in practice. This naturally raises a question which is unresolved in the literature so far as we are aware:

Q2: Does there exist a completely data driven estimator which does not depend on any unknown parameters of the problem and yet achieves MSE scaling like $\widetilde{O}(\tfrac{\textbf{V}}{\sqrt{N}})$, thus being simultaneously minimax rate optimal over V and $n$?

In Theorem 2.7 of this paper we answer this question in the affirmative by constructing such a fully data driven estimator.

The rest of the paper is organised as follows. In Section 2, we state our main results. Then in Section 3, we discuss connections of our results with some recent works and also present simulation studies which support and verify our main theorems. The proofs of our main results involve sharp bounds on the Gaussian widths (see (2.1) in Section 2.2.1) for some special classes of matrices. We obtain these bounds based on a generic approach which we detail in Section 4. The next five sections describe the proofs of our main theorems and intermediate results. Section 10 is the appendix which contains proofs of some auxiliary results.

Instructions for the reader

In all the proofs of our results from Section 5 onwards, we will use $\mathrm{TV}(\cdot)$ to denote the unnormalized version of (1.1). More precisely, for a $n\times n$ matrix $\theta$ we denote

[TABLE]

We adopt this convention because we believe it is easier to read and interpret the proofs with the unnormalized definition while it is instructive to use the normalized version for our theorems to facilitate interpretation of the risk bounds as a function of the sample size $N=n^{2}$. Also we will generically use $V$ to denote the unnormalized total variation whereas in Sections 1–3 we use bold $\mathbf{V}$ to denote the corresponding normalized total variation. In all our theorems presented in the next section we use bold $\mathbf{V}^{*}$ to denote $\mathrm{TV}_{\mathrm{norm}}(\theta^{*})$ where $\theta^{*}$ is the underlying true matrix and in all our proofs we use $V^{*}=\mathrm{TV}(\theta^{*})$ for the corresponding unnormalized version.

Acknowledgements: This research was supported by a NSF grant and an IDEX grant from Paris-Saclay. We thank the anonymous referees for extremely detailed comments and suggestions about the paper. These comments and suggestions have helped us to a great extent to improve our article.

2 Main Results

2.1 Constrained TVD

Our first result states a risk bound of $\widehat{\theta}_{\textbf{V}}$ under the bounded variation constraint.

Theorem 2.1**.**

Let $\theta^{*}$ be an arbitrary $n\times n$ matrix and $N=n^{2}$. Suppose the tuning parameter is chosen such that $\mathbf{V}\geq\mathbf{V}^{*}$. Then the following risk bound is true for a universal constant $C>0$:

[TABLE]

Remark 2.1**.**

The above result is similar to the $\ell_{1}$ bound of hutter2016optimal, the difference being the above risk bound holds for the constrained TVD estimator while the existing result of hutter2016optimal holds for the penalized estimator. For any sequence of $\mathbf{V}>0$ (possibly growing with $n$ although the canonical scaling is when $\mathbf{V}=O(1)$), the minimax lower bound results (mentioned earlier) of sadhanala2016total now imply the minimax rate optimality (up to log factors) of the constrained TVD estimator $\widehat{\theta}_{\mathbf{V}}$ over the parameter space $\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}_{\mathrm{norm}}(\theta)\leq\mathbf{V}\}$.

Remark 2.2**.**

As is made clear in Section 5, our technique for proving Theorem 2.1 is completely different from the technique used to prove the result of hutter2016optimal. While they analyze the properties of the pseudo-inverse of the edge incidence matrix $D,$ our proof relies on computing relevant Gaussian widths by recursive partitioning. Moreover, ingredients and ideas from this proof are also used crucially in the proofs of our other results.

2.2 Adaptive risk bound

Now we come to the main result of this paper which is about proving adaptive risk bounds for $\theta^{*}$ which are piecewise constant on at most $k$ axis aligned rectangles where $k$ is a positive integer much smaller than $N.$ We call a subset $R\subset L_{n}$ a (axis aligned) rectangle if it is a product of two intervals. For a generic rectangle $R=([a,b]\cap\mathbb{N})\times([c,d]\cap\mathbb{N})$, we define $\mathrm{n}_{\mathrm{row}}(R)$ and $\mathrm{n}_{\mathrm{col}}(R)$ to be the cardinalities of $[c,d]\cap\mathbb{N}$ and $[a,b]\cap\mathbb{N}$ respectively. In words, $\mathrm{n}_{\mathrm{row}}(R)$ and $\mathrm{n}_{\mathrm{col}}(R)$ are simply the numbers of rows and columns of $R$ respectively if one views $R$ as a two-dimensional array of points. Then we define its aspect ratio to be $A(R)\coloneqq\max\{\frac{\mathrm{n}_{\mathrm{row}}(R)}{\mathrm{n}_{\mathrm{col}}(R)},\frac{\mathrm{n}_{\mathrm{col}}(R)}{\mathrm{n}_{\mathrm{row}}(R)}\}$. For a given matrix $\theta\in\mathbb{R}^{n\times n}$ we define $k(\theta)$ to be the cardinality of the minimal partition of $L_{n}$ into rectangles $R_{1},\dots,R_{k(\theta)}$ such that $\theta$ is constant on each of the rectangles. Next we state our main result for the 2D TVD estimator.

Theorem 2.2**.**

Let $\theta^{*}\in\mathbb{R}^{n\times n}$ be the underlying true matrix with $\mathrm{TV}_{\mathrm{norm}}(\theta^{*})>0$ and $R_{1}^{*},\dots,R_{k(\theta^{*})}^{*}$ be its rectangular level sets which form a partition of the 2D grid $L_{n}.$ In addition, suppose the rectangles $R_{i}^{*}$ have bounded aspect ratio, that is there exists a constant $c>0$ such that $\max_{i\in[k(\theta^{*})]}A(R_{i}^{*})$ $\leq c$. Then we have the following risk bound:

[TABLE]

Here $C$ is a constant that only depends on $c.$

Remark 2.3**.**

Theorem 2.2 is really a statement about an ideally tuned constrained TVD estimator. One way to interpret it is that if the tuning parameter $\mathbf{V}$ is chosen such that $(\mathbf{V}-\mathbf{V}^{*})^{2}\leq C\sigma^{2}(\log\mathrm{e}n)^{9}\frac{\>k(\theta^{*})^{5/4}}{N^{3/4}}$ then the $\widetilde{O}(\frac{\>k(\theta^{*})^{5/4}}{N^{3/4}})$ rate of convergence holds.

Remark 2.4**.**

One consequence of the above theorem is that when $k(\theta^{*})=O(1)$ then the ideally tuned TVD estimator attains a $\widetilde{O}(N^{-3/4})$ rate. This rate is faster than the $\widetilde{O}(N^{-1/2})$ rate that is available in the literature. Our main focus here has been to attain the right exponent for $N$. The exponent of $k(\theta^{*})$ and $\log n$ may not be optimal. Since the current proof of this theorem is fairly involved technically, obtaining the best possible exponents of $k(\theta^{*})$ and $\log n$ is left for future research endeavors. See Section 3 for more discussions about the proof of the above theorem and comparisons with existing results.

Remark 2.5**.**

We think a bounded aspect ratio condition would actually be necessary for the $O(N^{-3/4})$ rate to hold in the above theorem; see Section 3.4 for more on this issue.

A natural question is whether our upper bound in Theorem 2.2 is tight. Our next theorem says that, in the low $\sigma$ limit, the $N^{-3/4}$ rate is not improvable even if $k(\theta^{*})=2.$

Theorem 2.3**.**

Let $\theta^{*}_{ij}=1$ if $j>n/2$ and [math] otherwise. Thus, $\theta^{*}$ is of the following form:

[TABLE]

Clearly $k(\theta^{*})=2.$ In this case, we have a lower bound to the risk of the ideally constrained TVD estimator.

[TABLE]

Here $c>0$ is a universal constant.

2.2.1 Gaussian width bounds

Proving Theorem 2.2 and Theorem 2.3 requires upper and lower bounds on the Gaussian width of a certain family of matrices as we now explain. The Gaussian width of a set $K\subset\mathbb{R}^{n}$ is defined as

[TABLE]

where $Z=Z_{n}\sim N(0_{n},I)$ and $\langle\cdot\,,\cdot\rangle$ is the usual Euclidean inner product between two vectors. We use $B_{m,n}(t)$ to denote the usual Euclidean ball of radius $t$ in $\mathbb{R}^{m\times n}$. For any $A\subset\mathbb{R}^{n\times n}$ we denote the smallest cone containing $A$ by ${\rm Cone}(A)$ and the closure of $A$ by ${\rm Closure}(A)$. The tangent cone $T(\theta^{*})\subset\mathbb{R}^{n\times n}$ at $\theta^{*}$ with respect to the closed convex set $K^{*}\coloneqq\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}_{\mathrm{norm}}(\theta)\leq\mathbf{V}^{*}\}$ is defined as follows:

[TABLE]

By definition, $T_{K^{*}}(\theta^{*})$ is a closed convex cone. Informally, $T_{K^{*}}(\theta^{*})$ represents all directions in which one can move infinitesimally from $\theta^{*}$ while still remaining in $K^{*}$.

Roughly speaking, the problem of bounding the MSE from both directions is equivalent to bounding the square of \mathcal{GW}\big{(}T_{{K}^{*}}(\theta^{*})\cap B_{n\times n}(1)\big{)} when $\theta^{*}$ is a piecewise constant matrix on rectangles. The precise connection of MSE to Gaussian widths is detailed in Section 6 where the proofs of Theorem 2.2 and Theorem 2.3 are also given. This connection prompts us to investigate how these tangent cones look like in the first place. The major technical contribution of this paper is to give upper and lower bounds on the Gaussian width of the tangent cone at a piecewise constant matrix which we encapsulate in the following two results.

Proposition 2.4**.**

Let $\theta\in\mathbb{R}^{n\times n}$ be a given matrix and $R_{1},\dots,R_{k(\theta)}$ be its rectangular level sets which form a partition of the 2D grid $L_{n}.$ In addition, let us assume that the rectangles $R_{i}$ have bounded aspect ratio, that is there exists a constant $c>0$ such that $\max_{i\in[k]}A(R_{i})\leq c$. Let $K\coloneqq\{v\in\mathbb{R}^{n\times n}:\mathrm{TV}_{\mathrm{norm}}(v)\leq\mathrm{TV}_{\mathrm{norm}}(\theta)\}$ and $T_{K}(\theta)$ be the tangent cone at $\theta$ with respect to $K.$ Then there is a universal constant $C>0$ such that

[TABLE]

Proposition 2.5**.**

Consider $\theta^{*}$ which is piecewise constant on two rectangles and is of the following form:

[TABLE]

Then, there exists a universal constant $c>0$ such that we have the following lower bound:

[TABLE]

The proofs of Proposition 2.4 and Proposition 2.5 are given in Sections 8 and 7 respectively.

It should be mentioned here that bounding the Gaussian width of the tangent cone is a fundamental task in a different but related problem of signal recovery from a given number of measurements; see ChandraFOCS and amelunxen2014living. Matrix recovery using 2D Total Variation has been studied in the signal processing literature; see for instance cai2015guarantees, genzel2020ell1 and kabanava2014robust. Our bounds on the Gaussian widths given in Proposition 2.4, Proposition 2.5 and Theorem 2.6 (see below) appear to be new and are potentially of independent interest as stand alone results. Especially our use of optimized partitioning schemes (see Section 8.6 for details) in the proof of Proposition 2.4 can be a useful strategy to attack other problems of similar flavor. See also Section 3.2 for further discussion on the novelty of our proof.

2.2.2 Impossibility of adaptation to non rectangular level sets

Theorem 2.2 shows that the $O(N^{-3/4})$ rate is achievable when $\theta^{*}$ is piecewise constant on a few rectangles. A question arises here as to what rate is achievable when $\theta^{*}$ is piecewise constant but the level sets are not rectangular. The following theorem says that for a simple matrix $\theta^{*}$ whose level sets are triangular, the $\widetilde{O}(N^{-1/2})$ rate cannot be improved.

Theorem 2.6**.**

Consider the signal matrix $\theta^{*}\coloneqq\mathbb{I}\{i+j>n\}$. Then, there exists a universal constant $c>0$ such that we have the following lower bound:

[TABLE]

Further, this implies a lower bound to the risk of the ideally constrained TVD estimator as follows:

[TABLE]

Here $c>0$ is a universal constant.

Remark 2.6**.**

The proof of the above theorem should be extendable when $\theta^{*}$ is indicator of a circle or a regular $n$ sided ($n>4$) polygon or any other shape which is sufficiently non rectangular. See Remark 7.1 for more on this issue. Therefore, it seems that the rectangular shape of the level sets is crucial for the faster $\widetilde{O}(N^{-3/4})$ rate to hold.

2.3 Tuning free TVD

We now state our final result which relates to the question we posed about removing the tuning parameter and still retaining a risk bound which is essentially the same as in Theorem 2.1. Choosing the tuning parameter is an important issue in applying the TVD methodology for denoising. The usual way out is to to do some form of cross validation. There are some proposals available in the literature; see solo1999selection, osadebey2014optimal, langer2017automated. Soon after we uploaded our paper, a different tuning parameter free method appeared in ortelli2019oracle which also achieves the optimal worst case $\widetilde{O}(\mathbf{V}/\sqrt{N})$ rate of convergence. See Section 3.3 for a comparison of our method with the one proposed in ortelli2019oracle.

Our goal here is to construct a tuning parameter free estimator of $\theta^{*}$ which adapts to the true value of $\mathrm{TV}_{\mathrm{norm}}(\theta^{*}).$ The inspiration for this task comes from chatterjee2015high where the author gives a general recipe to construct tuning parameter free estimators in Gaussian mean estimation problems when the truth is known to have small value of some known norm. Even though the total variation functional is not a norm but a seminorm, the general idea in chatterjee2015high can be extended as we will show. However, the estimator of chatterjee2015high is a randomized estimator whereas in our case we construct a non randomized version. The following is a description of our tuning free estimator.

Let 1 denote the $n\times n$ matrix consisting solely of ones. For any matrix $\theta\in\mathbb{R}^{n\times n},$ we define $\overline{\theta}\coloneqq\frac{1}{n^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}\theta[i,j]$ to be the mean of $\theta$. Define the estimator

[TABLE]

where $\widehat{\sigma}$ is an estimator of $\sigma$ defined as follows:

[TABLE]

The intuition behind the estimator defined above is as follows. The estimation of $\theta^{*}$ is done by estimating the two orthogonal parts $\overline{\theta^{*}}\>\textbf{1}$ and $\theta^{*}-\overline{\theta^{*}}\textbf{1}$ separately. The first part is estimated by $\overline{y}\>\textbf{1}$. To estimate $\theta^{*}-\overline{\theta^{*}}\textbf{1},$ we use a Dantzig Selector type (see candes2007dantzig) version of the TVD estimator, which computes a zero mean matrix with the least total variation subject to being within a Euclidean ball of a suitable radius around the centered data matrix $y-\overline{y}\,\mathbf{1}$. A good choice of this radius actually depends on the true $\sigma$ and hence as an intermediate step, we have to estimate $\sigma$ in the process which is denoted by $\widehat{\sigma}.$ The main idea behind our construction of $\widehat{\sigma}$ here is the fact that $\mathrm{TV}_{\mathrm{norm}}(\theta^{*})$ is small compared to $\mathrm{TV}_{\mathrm{norm}}(Z)$ and hence $\mathrm{TV}_{\mathrm{norm}}(y)=\mathrm{TV}_{\mathrm{norm}}(\theta^{*}+\sigma Z)$ approximately equals $\sigma\mathrm{TV}_{\mathrm{norm}}(Z).$ We can then use concentration properties of the $\mathrm{TV}_{\mathrm{norm}}(Z)$ statistic to show that $\tfrac{\mathrm{TV}_{\mathrm{norm}}(Z)}{\mathbb{E}\mathrm{TV}_{\mathrm{norm}}(Z)}$ is approximately equal to $1.$ The following theorem supplies a risk bound for $\widehat{\theta}_{\mathrm{notuning}}.$

Theorem 2.7**.**

We have the following risk bound for our tuning free estimator:

[TABLE]

where $C$ is a universal constant.

Remark 2.7**.**

Note that the above bound is meaningful only when $\lim_{N\to\infty}\tfrac{\mathbf{V}^{*}}{\sqrt{N}}=0$. Therefore in this regime, $\tfrac{(\mathbf{V}^{*})^{2}}{N}$ is a lower order term. Thus, Theorem 2.7 basically says that the MSE of $\widehat{\theta}_{\mathrm{notuning}}$, up to multiplicative log factors and an additive factor $\tfrac{\sigma}{\sqrt{N}}$, scales like $\tfrac{\mathbf{V}^{*}}{\sqrt{N}}$. In light of Remark 2.1 we can say that $\widehat{\theta}_{\mathrm{notuning}}$ is minimax rate optimal (up to log factors) over $\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}_{\mathrm{norm}}(\theta)\leq\mathbf{V}\}$, simultaneously for any sequence of V (depending on $n$) which is bounded below by a constant and above by $\sqrt{N}$. To the best of our knowledge, this is the first result demonstrating such an estimator which is completely tuning free.

3 Comparison with existing results, simulation studies and discussions

To place our theorems in context, it is worthwhile to compare and relate our results with a couple of recent papers. We also discuss some issues related to our results.

3.1 Comparison with hutter2016optimal

Let us compare our risk bound in Theorem 2.2 to the adaptive risk bound (Theorem 1.1) of hutter2016optimal when the truth $\theta^{*}$ is piecewise constant on a few axis aligned rectangles. Both of these theorems prove statements about tuned TVD estimators. Considering the very simple case when $\theta^{*}$ is of the following form:

[TABLE]

we have already mentioned in Section 1 that $\|D\theta^{*}\|_{0}=O(\sqrt{N}).$ Thus, Theorem 1.1 gives us an upper bound on the MSE scaling like $\widetilde{O}(1/\sqrt{N})$ whereas our Theorem 2.2 gives a faster rate of convergence scaling like $\widetilde{O}(1/N^{3/4}).$ More generally, if $\theta^{*}$ is piecewise constant on $k$ axis aligned rectangles with bounded aspect ratio and roughly equal size, it can be checked that $\|D\theta^{*}\|_{0}\approx\sqrt{k^{*}N}.$ This means that Theorem 1.1 gives us an upper bound on the MSE scaling like $\widetilde{O}(\sqrt{k^{*}/N}).$ Compare this to Theorem 2.2 which gives a rate of convergence scaling like $\widetilde{O}((k^{*})^{5/4}/N^{3/4}).$ Thus, in the small $k^{*}$ regime when $k^{*}<N^{1/3}$, Theorem 2.2 provides a faster rate of convergence. This is the main contribution of this paper and to the best of our knowledge is the first of its kind in the literature.

3.2 Comparison with guntuboyina2017adaptive

As mentioned in Section 1, one of our motivating factors behind investigating adaptivity of the 2D TVD estimator was its success in optimally estimating piecewise constant vectors in the 1D setting. Theorem $2.2$ in guntuboyina2017adaptive gives a $\widetilde{O}(k^{*}/n)$ rate for the ideally tuned constrained 1D TVD estimator when the truth $\theta^{*}$ is piecewise constant with $k^{*}$ pieces or blocks and each block satisfies a certain minimum length condition. In a sense, our Theorem 2.2 is a natural successor, giving the corresponding result in the 2D setting. Our bounded aspect ratio condition is the 2D version of the minimum length condition. A consequence of Theorem 2.2 and Theorem 2.3 is that, in contrast to the 1D setting, the ideally tuned constrained TVD estimator can no longer obtain the oracle rate of convergence $\widetilde{O}(k^{*}/n)$ in the 2D setting.

The proof of Theorem $2.2$ in guntuboyina2017adaptive was done by bounding the Gaussian widths of certain tangent cones. Our proof of Theorem 2.2 also adopts the same strategy and precisely characterizes the tangent cone $T_{K(V^{*})}(\theta^{*})$ (defined in (2.2)) for piecewise constant $\theta^{*}$ and then bounds its Gaussian width. The main idea in guntuboyina2017adaptive was to observe that any unit norm element of the tangent cone is nearly made up of two monotonic blocks in each constant block of $\theta^{*}.$ Then the available metric entropy bounds for monotone vectors were used to bound the Gaussian width. A crucial ingredient in this proof is the well-known fact that any univariate function of bounded variation has a canonical representation as a difference of two monotonic functions. However, it is not clear at all how to adapt such a strategy to the 2D setting. In particular, it is not nearly as natural and convenient to express a matrix of bounded variation as a difference of two bi-monotone matrices. Our computation of Gaussian width of the tangent cone is therefore essentially two dimensional and involves judicious recursive partitioning in both dimensions. We believe that our Gaussian width computations, especially the proof of Proposition 8.9, consist of new techniques and are potentially useful for problems of similar flavor.

3.3 Comparison with ortelli2019oracle

At the latter stages of preparation of this manuscript we became aware of an independent work by ortelli2019oracle which is related to our manuscript. In ortelli2019oracle, the authors give a general technique to derive slow ($\ell_{1}$) and fast ($\ell_{0}$) rates for penalized TVD estimators and its square root version on general graphs. Thus, there seems to be two routes for obtaining fast rates for TVD. One goes through the route of bounding Gaussian width of an appropriate tangent cone to derive fast rates for the constrained TVD estimator; as done here in this manuscript as well as in guntuboyina2017adaptive. The other route; followed by hutter2016optimal and generalized by ortelli2019oracle is based on bounding the so called compatibility factor. ortelli2019oracle show how to bound this compatibility factor for specific graphs such as the $1d$ grid graph and the $1d$ cycle graph. To the best of our knowledge, bounding the compatibility factor for piecewise constant functions on axis aligned rectangles for a $2d$ grid remains an open problem. Thus, as far as we are aware, the work in this manuscript proving fast rates of convergence on $2d$ grid graph for piecewise constant functions on axis aligned rectangles is the first of its type in the literature.

The work in ortelli2019oracle also proposes a general technique to obtain slow rates for a square root version of the TVD estimator. Similar to our paper, ortelli2019oracle also considers the case when the noise variables are i.i.d. Gaussian. The advantage of using this square root version is that the tuning parameter does not need to depend on the unknown parameter $\sigma$. While the theoretically recommended choice of the tuning parameter $\lambda$ in (ortelli2019oracle, Corollary 4.13) does not depend on the noise variance $\sigma^{2}$, there is however the presence of an unspecified large universal constant $C$. It is not clear to us whether this $C$ can be explicitly specified. On the other hand, our tuning free estimator is explicitly specified and involves no unknown constants. We think the analysis of our tuning free estimator is also reasonably clean with the sources of the various possible errors made transparent in the proof. This is why we believe that our tuning free estimator provides a theoretically valid and possibly useful alternative to the square root regularization approach. Just to be clear, we are not claiming any optimality of our tuning free method, our intention is to demonstrate one theoretically valid way to obtain a minimax rate optimal tuning free estimator.

3.4 Necessity of bounded aspect ratio condition in theorem 2.2

We think a bounded aspect ratio condition would actually be necessary for the $O(N^{-3/4})$ rate to hold in Theorem 2.2. For instance, consider the sequence of matrices $\theta^{*}$ such that $\theta^{*}[i,j]=\mathbb{I}\{j=n\}.$ Clearly, the rectangular level sets of the sequence of matrices $\theta^{*}$ do not satisfy the bounded aspect ratio condition. By an argument similar to the one used to prove our lower bound results in Theorem 2.3 and Theorem 2.6, one can show that the ${\rm MSE}(\widehat{\theta}_{\mathbf{V}^{*}},\theta^{*})\geq cN^{-1/2}.$ We have also verified this scaling of the MSE in our numerical experiments.

The bounded aspect ratio condition says that the rectangular level sets of $\theta^{*}$ should not be too skinny or too long. In our proof, the bounded aspect ratio is needed for similar reasons as a minimum length condition is needed for the length of the pieces in the 1D setting; see guntuboyina2017adaptive, dalalyan2017tvd.

3.5 On obtaining the oracle rate for piecewise constant signals

In light of Theorem 2.2 and Theorem 2.3 we can say the following statement. When the truth $\theta^{*}$ is piecewise constant on $k^{*}$ axis aligned rectangles, the TVD estimator cannot attain the oracle rate of convergence scaling like $O(k^{*}/N).$ The question that now arises is whether there exists any estimator which attains the $\widetilde{O}(k^{*}/N)$ rate of convergence for all piecewise constant truths as well as the minimax rate $\widetilde{O}{(\mathbf{V}^{*}/\sqrt{N})}$? Furthermore, can this estimator be chosen so that it is computationally efficient? These questions led us to examine decision tree estimators which are different from the TVD type estimators. We would like to point out here that in the paper chatterjeegoswamiadaptive we have been able to demonstrate computationally efficient estimators which attain both the aforementioned goals.

Apart from chatterjeegoswamiadaptive, some other recent papers have also sprung up which target piecewise rectangular signals. The papers ortelli2019tvd and fang2019multivariate study a different version of the TVD estimator which is also termed as the Hardy Krause estimator. As far as we understand, this estimator is well suited for estimating piecewise rectangular signals as it actually fits piecewise rectangular estimates. For a general signal with $k^{*}$ rectangular pieces (with some regularity conditions on the rectangular pieces), the rate proved by ortelli2019tvd is $\widetilde{O}(({k^{*}})^{3/2}/N)$ which is better than \widetilde{O}\big{(}({k^{*}})^{5/4}/N^{3/4}\big{)}. Notice that the $\widetilde{O}(({k^{*}})^{3/2}/N)$ rate still does not match the near oracle rate $\widetilde{O}({k^{*}}/N)$ which has been obtained in chatterjeegoswamiadaptive.

3.6 Constrained vs Penalized

In this paper, we have focussed on the constrained version of the 2D TVD estimator. As mentioned in the introduction, the penalized version is also quite popular. In the low $\sigma$ limit, it can be proved that the constrained estimator $\widehat{\theta}_{\mathbf{V}}$ with $\mathbf{V}=\mathbf{V}^{*}=\mathrm{TV}_{\mathrm{norm}}(\theta^{*})$ is better than the penalized estimator for every deterministic choice of $\lambda.$ More precisely, we have for all $\lambda\geq 0$,

[TABLE]

The above inequality follows from the results of oymak2013sharp as described in Section 5.2 in guntuboyina2017adaptive. Since our main question here is whether faster/adaptive rates are possible for piecewise constant matrices, it is therefore natural to first study the constrained version with ideal tuning. A possible next step is to investigate whether a similar $N^{-3/4}$ rate is atttained by the penalized TVD estimator and if so, for what range of the tuning parameter $\lambda$.

3.7 Simulation studies

We consider three distinct sequences of matrices to facilitate comparison. We consider the simplest piecewise constant matrix $\theta^{\mathrm{two}}\in\mathbb{R}^{n\times n}$ where $\theta^{\mathrm{two}}\coloneqq\mathbb{I}\{j>n/2\}.$ Hence $\theta^{\mathrm{two}}$ just takes two distinct values. The next matrix $\theta^{\mathrm{four}}$ is a block matrix with four constant blocks.

[TABLE]

Finally, we also consider a $n\times n$ matrix $\theta^{\mathrm{worst}}\coloneqq\mathbb{I}\{i+j>n\}$. Clearly, $\theta^{\mathrm{worst}}$ does not have a block constant structure. For the matrix $\theta^{\mathrm{worst}}$ we incur the worst case rate $\widetilde{O}(N^{-1/2})$ as shown in Theorem 2.6; hence the name.

The dependence of the MSE with $N=n^{2}$ can be experimentally checked as follows. We can estimate the MSE for a fixed $n$ by Monte Carlo repetitions and then iterate this for a grid of $n$ values. We then plot log of the estimated MSE with log $N$ and fit a least squares line to the plot. The slope of the least squares line then gives an indication of the correct exponent of $N$ in the MSE. Figure 1 is such a plot for the ideally tuned constrained TVD estimator.

In Figure 1, the risk is seen to be minimum for $\theta^{\mathrm{two}}$ followed by $\theta^{\mathrm{four}}$ and then $\theta^{\mathrm{worst}}.$ The slope for $\theta^{two}$ and $\theta^{\mathrm{four}}$ came out to be $-0.73$ and $-0.68.$ This agrees well with Theorem 2.2 and Theorem 2.3 which says that the MSE decays at the rate $n^{-0.75}$ upto log factors. For the matrix $\theta^{\mathrm{worst}}$ the slope turned out be $-0.52$ which is in agreement with the worst case $\widetilde{O}(N^{-1/2})$ rate given in Theorem 2.6.

To investigate the dependence of MSE with the number of rectangular level sets $k(\theta^{*})$, we took four matrices. The first two are $\theta^{\mathrm{two}},\theta^{\mathrm{four}}$ and the last two are obtained by further binary division so that the number of rectangular level sets is $8,16$ respectively. We normalized the matrices such that $\mathbf{V}^{*}=1.$ We fixed $n=800$ and did $50$ iterations for each of the four matrices. We then plotted $\log{\rm MSE}$ versus $\log_{2}k$ (see Figure 2) where $k=2,4,8,16.$ The slope of the least squares line we got is $0.81.$ This suggests that our exponent of $k$ ($=1.25)$ in the risk bound in Theorem 2.2 may not be optimal.

To assess the risk of our fully data driven estimator $\widehat{\theta}_{\mathrm{notuning}}$, we again consider the three matrices $\theta^{\mathrm{two}},\theta^{\mathrm{four}}$ and $\theta^{\mathrm{worst}}$ respectively. Figure 3 is a plot of log MSE versus $\log n.$

The simulations in Figure 3 suggest that our estimator has MSE decaying at a $O(1/\sqrt{N})$ rate for all three matrices. The slope of all three least squares lines are reasonably close to $-0.5.$ This matches the rate given in Theorem 2.7. However, our tuning free estimator does not seem to be adaptive to piecewise constant structure like the constrained TVD estimator with ideal tuning.

To investigate the dependence of the risk of our tuning free estimator on $\mathbf{V}^{*}$, for each of the three matrices $\theta^{\mathrm{two}}$, $\theta^{\mathrm{four}}$, $\theta^{\mathrm{worst}}$, we normalized the matrix such that $\mathbf{V}^{*}=1,2,\dots,10$. We fixed $n=200$ and did $50$ iterations for each $\mathbf{V}^{*}$ and each matrix. We then plotted log MSE versus log $\mathbf{V}^{*}$ (see Figure 4) and fitted a least squares line. The slopes for each of these three matrices came out to be $1.16,1.07,0.94$ respectively. This suggests that the right exponent of $V^{*}$ is $1$ and our risk bound has the right dependence on $\mathbf{V}^{*}.$

4 A generic approach towards bounding Gaussian widths

Let us recall from Section 2.2.1 that the Gaussian width of a set $K\subset\mathbb{R}^{n}$ is defined as

[TABLE]

where $Z=Z_{n}\sim N(0_{n},I)$ and $\langle\cdot\,,\cdot\rangle$ is the usual Euclidean inner product between two vectors. Our principal result in this section provides an upper bound on $\mathcal{GW}(K)$ in terms of the numbers and dimensions of its covering (linear) subspaces. This result executes and adapts the idea of chaining (see, e.g., (van2014probability, Theorem $5.24$)) to the case when the covering sets are linear subspaces of $\mathbb{R}^{n}$. To this end let us define, for any $\epsilon>0$, an $\epsilon$ subspace cover of $K$ to be any finite collection $\mathcal{S}$ of linear subspaces of $\mathbb{R}^{n}$ such that

[TABLE]

where ${\rm dist}(A,B)$ denotes the Euclidean distance between the sets $A$ and $B$. We denote by $\mathrm{diam}(K)$ the diameter of $K$ which we assume to be finite. Also for any $t>0$, we denote by $B_{n}(t)$ the $t$-Euclidean ball $\{\theta\in\mathbb{R}^{n}:\|\theta\|\leq t\}$ where $\|.\|$ is the Euclidean norm. We will often drop the subscript $n$ and just write $B(t)$ when the dimension is clear from the context.

Proposition 4.1** (Gaussian width bound).**

For every $\epsilon\in(0,\mathrm{diam}(K))$, let $\mathcal{S}_{\epsilon}$ be an $\epsilon$ subspace cover of $K$. Also let $k_{1}>k_{0}$ be integers with $k_{0}$ being the smallest integer satisfying $2^{-k_{0}}\geq\mathrm{diam}(K)$. Then we have

[TABLE]

*for some universal constant $C>0$. *

Proof.

For any $\theta\in K$ and integer $k$ such that $2^{-k}<\mathrm{diam}(K)$, let $\theta_{k}$ denote a point in $\mathcal{N}_{k}\coloneqq\cup_{S\in\mathcal{S}_{2^{-k}}}S$ such that

[TABLE]

Such a point always exists since $\mathcal{N}_{k}$ is a finite union of linear subspaces. When $\mathrm{diam}(K)\geq 2^{-k}$, on the other hand, we simply choose $\theta_{k}$ to be some fixed but arbitrary point $\theta_{K}$ in $K$. By definition, we thus have

[TABLE]

for all $k\in\mathbb{Z}$. Let us now write for every $\theta\in K$,

[TABLE]

so that

[TABLE]

The first term on the right hand side above is 0, whereas the third time is bounded by $\sqrt{n}2^{-k_{1}}$ in view of the Cauchy-Schwarz inequality, display (4.1) and the standard bound $\mathbb{E}\|Z\|\leq\sqrt{n}$. Therefore we can conclude the proof if we can show

[TABLE]

for every integer $k$ satisfying $2^{-k}<\mathrm{diam}(K)$. To this end observe that

[TABLE]

in view of (4.1) and $\theta_{k}-\theta_{k-1}\in\mathcal{M}_{k}$ where $\mathcal{M}_{k}\coloneqq\{S_{1}+S_{2}:S_{1}\in\mathcal{N}_{2^{-k}},S_{2}\in\mathcal{N}_{2^{-(k-1)}}\}$ is another finite collection of linear subspaces of $\mathbb{R}^{n}$. It is also clear from the definition that $|\mathcal{M}_{k}|\leq|\mathcal{N}_{2^{-k}}||\mathcal{N}_{2^{-(k-1)}}|$. All these observations bring us to the setting of:

Lemma 4.2** (Gaussian width for union of subspaces).**

Let $\mathcal{S}$ be a finite collection of linear subspaces of $\mathbb{R}^{n}$ and $\Theta=\cup_{S\in\mathcal{S}}S\subset\mathbb{R}^{n}.$ In words, $\Theta$ is the union of subspaces in $\mathcal{S}$. Then we have

[TABLE]

Using Lemma 4.2, we can immediately deduce that

[TABLE]

where

[TABLE]

Now we can assume without any loss of generality that $|\mathcal{S}_{2^{-(k-1)}}|\leq|\mathcal{S}_{2^{-k}}|$ as well as

[TABLE]

which finishes the proof of the proposition.

Let us now return to the proof of Lemma 4.2. Since $\Theta=t\,\Theta$, it follows from the definition of Gaussian widths that $\mathcal{GW}(\Theta\cap B(t))=t\,\mathcal{GW}(\Theta\cap B(1))$ meaning we only need to work with $t=1$. We will use the following lemma involving only one linear subspace:

Lemma 4.3**.**

For any linear subspace $S$ of $\mathbb{R}^{n}$ and $u\geq 0$, we have with probability at least $1-\exp(-\frac{u^{2}}{2}),$

[TABLE]

Proof.

We will use the well-known concentration inequality for Lipschitz functions of a Gaussian vector (see, e.g. (Ledoux01conc, Theorem 7.1)). First of all notice that the random variable $f(Z)\coloneqq\sup_{\theta\in S\cap B(1)}\langle Z,\theta\rangle$ is a Lipschitz function of $Z$ with Lipschitz constant $1$. It follows from the observation that, for any $z,z^{\prime}\in\mathbb{R}^{n}$ and $\theta\in B(1)$,

[TABLE]

where in the last but one step we used the Cauchy-Schwarz inequality. Therefore by the Gaussian concentration inequality mentioned in the beginning, we have for any $u\geq 0$

[TABLE]

Hence we can deduce the lemma upon showing that $\mathbb{E}f(Z)\leq\sqrt{\dim(S)}$. To this end notice that $f(Z)=\|P_{S}Z\|$ where $P_{S}$ is the orthogonal projector onto the subspace $S$. Therefore, $f(Z)^{2}$ is a chi squared random variable whose degree of freedom equals $\dim(S)$ whence we get

[TABLE]

Now, using a union bound followed by Lemma 4.3 we get

[TABLE]

Plugging in $u=\sqrt{2\>\log|\mathcal{S}|+v^{2}}$ we obtain

[TABLE]

Integrating the above tail bound finishes the proof of Lemma 4.2. ∎

Remark 4.1**.**

A general and perhaps more standard way of bounding the Gaussian width of a set is through Dudley’s entropy integral inequality (see Dudley67). In this approach one first finds a “good” covering set corresponding to any given radius $r$ for the underlying set to obtain upper bounds on covering numbers which then enter an integral (after being transformed appropriately) bounding the Gaussian width. Proposition 4.1 provides an alternative way when the covering sets are contained in finite unions of linear subspaces with comparable dimensions. For the purpose of the current article, this approach would save us some extraneous log factors in our bounds.

5 Proof of Theorem 2.1

We first set up some notations which would henceforth be used throughout the paper. For a positive integer $n$, we will denote the subset of positive integers $\{1,\dots,n\}$ by $[n]$. Recall that in all the proofs of our results, we are going to use $\mathrm{TV}$ to denote the unnormalized version of (1.1) as defined in (1.2). Also we will use $V$ for the unnormalized total variation instead of the bold $\mathbf{V}$ used for the corresponding normalized version.

Let us recall that the estimator $\widehat{\theta}_{V}$ is the least squares estimator on the set

[TABLE]

We will often drop the subscript $n$ and just write $K(V)$ when the dimension is clear from the context. Below we adopt the standard approach of using the basic inequality defining least squares estimators to reduce our problem to controlling Gaussian widths.

Lemma 5.1**.**

Under the same conditions as in the statement of Theorem 2.1 we have

[TABLE]

Proof.

Since $V\geq V^{*}$ we have the basic inequality $\|y-\widehat{\theta}_{V}\|^{2}\leq\|y-\theta^{*}\|^{2}$. Substituting $y=\theta^{*}+\sigma Z$ gives us

[TABLE]

where the last inequality follows because $\overline{\widehat{\theta}_{V}}=\overline{y}$ and 1 refers to the $n\times n$ matrix whose all elements equal 1. Now taking expectation on both sides of the above display and noting that

[TABLE]

finishes the proof. ∎

Let us define

[TABLE]

In view of Lemma 5.1, all we need is to evaluate the Gaussian width of the set $K^{0}(2V)$ to which end we will use Proposition 4.1. But for that we need to find “efficient” subspace covers of the set $K^{0}(V)$ corresponding to any distance $\epsilon$. Our next proposition will be crucial for this purpose. Below we denote, for any rectangular partition $P$ of $L_{n}$, the linear subspace of $\mathbb{R}^{n\times n}$ comprising only matrices that are constant on each (rectangular) block of $P$ by $S_{P}$.

Proposition 5.2**.**

For every $\eta,V>0$, there exist a set of rectangular partitions $\mathcal{P}(V,n,\eta)$ of $L_{n}$ (recall the definition from Section 2.2) and a universal constant $C>0$ such that

• •

For any $\theta\in K(V)$, there exists a partition $P\in\mathcal{P}(V,n,\eta)$ satisfying

[TABLE]

• •

Any partition $P\in\mathcal{P}(V,n,\eta)$ has number of (rectangular) blocks bounded by

[TABLE]

• •

The cardinality of $\mathcal{P}(V,n,\eta)$ is bounded as

[TABLE]

Before we prove this proposition, let us finish the proof of Theorem $2.1$ assuming it.

Proof of Theorem 2.1.

Throughout this proof, we will use $C$ to denote an unspecified but universal positive constant whose exact value may change from one line to the next. For any $\epsilon,V>0$, let $\eta_{\epsilon}\coloneqq\min(\frac{\epsilon^{2}}{2V\log n},\frac{\epsilon}{\sqrt{2}})$ and consider the set of rectangular partitions $\mathcal{P}(V,n,\eta_{\epsilon})$ given by Proposition 5.2. Next define a collection $\mathcal{S}_{\epsilon}$ of linear subspaces of $\mathbb{R}^{n\times n}$ as follows:

[TABLE]

By Proposition 5.2 it can be seen that $\mathcal{S}_{\epsilon}$ forms an $\epsilon$ subspace cover of $K(V)$ and hence of $K^{0}(V)$ as well. Also, from the second and third properties of $\mathcal{P}(V,n,\eta)$ we get

[TABLE]

We now have all the ingredients to apply Proposition 4.1 except for an upper bound on the diameter of $K^{0}(V)$. To this end we use Proposition 5.3 — which we are going to state in the next subsection — to deduce that $t\coloneqq{\rm diam}(K^{0}(V))\leq C\,V$. We thus obtain from Proposition 4.1, with $k_{0}=\lfloor-\log_{2}t\rfloor$ and $k_{1}=\lceil\log_{2}(\frac{n}{V})\rceil$,

[TABLE]

Theorem 2.1 now follows immediately from Lemma 5.1 ∎

5.1 Proof of Proposition 5.2

Given any partition $P$ of $L_{n}$ into rectangles, it is clear that the orthogonal projection of $\theta$ onto $S_{P}$, i.e., the unique matrix $\widehat{\theta}_{P}\in S_{P}$ satisfying $\|\theta-\widehat{\theta}_{P}\|={\rm dist}(\theta,S_{P})$, is constant on every rectangle $R$ of $P$ with the common value being the mean of $\theta_{|R}$ — the restriction of $\theta$ to $R$. Therefore, with $\bar{\theta}_{|R}$ denoting the mean of $\theta_{|R}$,

[TABLE]

where $1_{R}\in\mathbb{R}^{R}$ consists only of 1’s. Our next result provides a way to bound the squared Frobenius distance between $\theta_{|R}$ and $\bar{\theta}_{|R}1_{R}$ in terms of the total variation of $\theta_{|R}$. This result, which is a discrete analogue of the Gagliardo-Nirenberg-Sobolev inequality for compactly supported smooth functions, will be crucial for deriving the first condition stipulated in Proposition 5.2 for the particular partitioning scheme we are going to propose in this regard.

Proposition 5.3** (Discrete Gagliardo-Nirenberg-Sobolev Inequality).**

Let $\theta\in\mathbb{R}^{m\times n}$ and $\overline{\theta}\coloneqq\sum_{i=1}^{m}\sum_{j=1}^{n}\theta[i,j]/mn$ be the average of the elements of $\theta$. Then we have

[TABLE]

So in particular when $m=n$, we have

[TABLE]

Remark 5.1**.**

Although the Gagliardo-Nirenberg-Sobolev inequality is classical for Sobolev spaces (see, e.g., Chapter 12 in leoni2017first), we are not aware of any discrete version in the literature that applies to arbitrary matrices of finite size. Also it is not clear if the inequality in this exact form follows directly from the classical version.

Now we give a scheme for subdividing $\theta$ in multiple steps until the total variation of each of the resulting submatrices is bounded above by $\eta$.

**A greedy partitioning scheme:  **For convenience of we will assume that $n$ is an integer power of 2. The general $n$ can then be accommodated from the following observation. For any $t>0$, let $B_{n,n}(t)$ denote the $t$-Euclidean (Frobenius) ball in $\mathbb{R}^{n\times n}$ and consider $\theta\in K_{n}(V)\cap B_{n,n}(t)$ (recall from our proof of Theorem 2.1 that we actually bound $\mathcal{GW}(K_{n}(V)\cap B_{n,n}(CV))$ for some universal constant $C$). Now let $n^{\prime}$ denote the smallest integer power of 2 that is larger than or equal to $n$ and partition $\theta$ as

[TABLE]

where $\theta_{22}\in\mathbb{R}^{(n^{\prime}-n)\times(n^{\prime}-n)}$. Also define a $n^{\prime}\times n^{\prime}$ matrix $f(\theta)$ as

[TABLE]

where $\overleftarrow{M}$, for any matrix $M$, denotes the matrix obtained by reversing the order of its columns whereas $M\uparrow$ is obtained by reversing the order of its rows. It is clear from the definition that $f(\theta)\in K_{n^{\prime}}(4V)\cap B_{n^{\prime},n^{\prime}}(t)$ and also

[TABLE]

where $Z_{n^{\prime},n^{\prime}}\sim N(0_{n^{\prime}\times n^{\prime}},I)$.

Let us now describe the scheme which is of the same flavor as the breadth-first exploration of a quaternary tree. The root node of the tree represents $L_{n}$ and the nodes at any level (or depth) $i\in[\log_{2}n]$ represent (disjoint) rectangles of side-length $n2^{-i}$ with the property that the leaves of the tree truncated at level $i$ form a partition of $L_{n}$. Given level $i-1$, the $i$-th level is constructed (or explored) as follows. For every leaf, i.e., rectangle $R$ at level $i-1$ satisfying $\mathrm{TV}(\theta_{|R})>\eta$, we add four children of $R$, namely $R_{11},R_{12},R_{21}$ and $R_{22}$, to the tree where

[TABLE]

and $\mathrm{n}_{\mathrm{row}}(R_{11})=\mathrm{n}_{\mathrm{col}}(R_{11})=\mathrm{n}_{\mathrm{row}}(R)/2$. If the set of such leaves is empty or if $i-1=\log_{2}n$, we stop.

Let us denote the final rectangular partition of $L_{n}$ obtained by applying the $(\mathrm{TV},\eta)$ scheme to $\eta$ as $P_{\theta,\eta}$ and the set of partitions $\{P_{\theta,\eta}:\theta\in\mathcal{\mathbb{R}}^{n\times n},\mathrm{TV}(\theta)\leq V\}$ as $\mathcal{P}(V,n,\eta)$. In our next result we verify that $\mathcal{P}(V,n,\eta)$ satisfy the last two properties stipulated in Proposition 5.2.

Lemma 5.4**.**

There exists a universal constant $C>0$ such that for any $\theta\in\mathbb{R}^{n\times n}$ and $\eta>0$, we have

[TABLE]

Furthermore, for any $V>0$ we have

[TABLE]

Proof.

The basic idea of the proof hinges on super-additivity of the $\mathrm{TV}$ functional over disjoint rectangles. Let $n_{i}$ denote the number of leaves in the tree formed by the scheme truncated at level $i$. In other words, $n_{i}$ is the cardinality of the partition $P_{i}$ formed by the rectangles corresponding to the leaves of the tree truncated at level $i$. Also let $s_{i}$ denote the number of leaves $R$ at level $i$ satisfying $\mathrm{TV}(\theta_{|R})>\eta$. Clearly $n_{0}=1$ and $n_{i+1}=n_{i}+3s_{i}$. Notice that, due to super-additivity of the $\mathrm{TV}$ functional, we must have

[TABLE]

This implies in particular that

[TABLE]

Since $i\leq\log_{2}n$ by construction, it then follows

[TABLE]

Next we bound the number of possible partitions $P_{\theta,\eta}$ when $\mathrm{TV}(\theta)\leq V$. The number of distinct ways of adding leaves at level $i+1$ is at most \big{(}1+3i\frac{V}{\eta}\big{)}^{\frac{V}{\eta}} in light of the displays (5.4) and (5.5). Therefore

[TABLE]

for some universal constant $C>0$. ∎

With Lemma 5.4 and Proposition 5.3 in hand, we are now in a position to finish the proof of Proposition 5.2.

Proof of Proposition 5.2.

For any given $\theta\in K(V)$, run the $(\mathrm{TV},\eta)$ greedy scheme to obtain the partition $P_{\theta,\eta}.$ Within every rectangle of the partition $P_{\theta,\eta}$ the total variation of $\theta$ is at most $\eta$. Also, the number of rectangles in $P_{\theta,\eta}$ is at most $1+C\frac{V}{\eta}\log n$. Then by Proposition 5.3 and (5.3) we can conclude

[TABLE]

Also, by Lemma 5.4, as $\theta$ varies in $K(V)$, the number of distinct partitions $P_{\theta,\eta}$ that can be obtained is bounded by $\frac{V}{\eta}\log n$. This finishes the proof. ∎

Finally it remains to give the proof of Proposition 5.3.

Proof of Proposition 5.3.

For any $(i,j)\in[m]\times[n]$, we have

[TABLE]

where $\theta[i,0]=\theta[0,j]=\overline{\theta}$ for all $(i,j)\in[m]\times[n]$. Summing this over all $i$ and $j$ we get

[TABLE]

Here the total variation ${\mathrm{TV}}_{\mathrm{row}}(\theta)$ along rows is defined as

[TABLE]

and ${\mathrm{TV}}_{\mathrm{col}}(\theta)\coloneqq{\mathrm{TV}}_{\mathrm{row}}(\theta^{T})$. Now let us try to bound $|\theta[i,1]-\overline{\theta}|$.

[TABLE]

Hence

[TABLE]

Similarly

[TABLE]

Plugging these bounds into the last expression in (5.1), we get

[TABLE]

6 Proofs of Theorem 2.2 and Theorem 2.3

We first describe the precise connection between MSE and Gaussian widths. Recall that use $B_{m,n}(t)$ to denote the usual Euclidean ball of radius $t$ in $\mathbb{R}^{m\times n}$. The statistical dimension of a closed convex cone $K\subset\mathbb{R}^{N}=\mathbb{R}^{n\times n}$ is defined as

[TABLE]

and $\Pi_{K}(Z):=\mathop{\rm argmin}_{u\in K}\|Z-u\|^{2}$ is the Euclidean projection of $Z$ onto $K$. The terminology of statistical dimension is due to amelunxen2014living and we refer the reader to this paper for many properties of the statistical dimension. The statistical dimension $\delta(K)$ is closely related to the Gaussian width of $K\cap B_{n,n}(1).$ It has been shown in (amelunxen2014living, Proposition 10.2) that

[TABLE]

for every closed convex cone $K$.

The connection of the statistical dimension of tangent cones to the risk of $\widehat{\theta}$ is the content of the following result due to bellec2018sharp.

Theorem 6.1** ( bellec2018sharp).**

Suppose $Y\sim N(\theta^{*},\sigma^{2}I)$ for some $\theta^{*}\in\mathbb{R}^{N}$. Then

[TABLE]

Another result that is of use to us is the following result of oymak2013sharp (Theorem $2.1$). It says that the upper bound provided in Theorem 6.1 is essentially tight. Recall from Section 2.2.1 that $K^{*}=\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}(\theta)\leq\mathrm{TV}(\theta^{*})\}.$

Theorem 6.2** ( oymak2013sharp ).**

[TABLE]

Remark 6.1**.**

To clarify, Theorem $2.1$ in oymak2013sharp actually says that

[TABLE]

Here $Z$, as usual, refers to a matrix of independent $N(0,1)$ entries, $\mathrm{Polar}(T_{K^{*}}(\theta^{*}))$ refers to the Polar Cone of  $T_{{K}^{*}}(\theta^{*})$ and $\mathrm{dist}$ refers to the Euclidean Distance between two sets. Letting $K$ denote a general cone and $\Pi_{K}$ denote the Euclidean projection operator onto $K$, the standard Pythagorean Theorem for cones implies

[TABLE]

Also, it holds that $\|\Pi_{K}(Z)\|=\sup_{\theta\in K:\|\theta\|\leq 1}\langle Z,\theta\rangle$. A proof of the above fact is available in Lemma $A.3$ in chatterjee2019adaptive. Theorem 6.2 now follows from applying the above facts to Theorem $2.1$ in oymak2013sharp and then using the elementary inequality $\mathbb{E}X^{2}\geq(\mathbb{E}X)^{2}.$

In light of the above facts and armed with Proposition 2.4 and Proposition 2.5 we are now ready to prove Theorem 2.2 and Theorem 2.3 respectively.

Proof of Theorem 2.2.

Theorem 6.1 along with (6.1) gives us

[TABLE]

With $V^{*}=\mathrm{TV}(\theta^{*})>0$, define

[TABLE]

By definition, $\mathrm{TV}(\theta)=V$ and $\theta$ is piecewise constant on the same partition of $L_{n}$ as is $\theta^{*}$. By Proposition 5.3 we can assert that

[TABLE]

Therefore, in view of (6.2), we obtain

[TABLE]

where we have also used Proposition 2.4 and the fact that $k(\theta)=k(\theta^{*}).$ ∎

Proof of Theorem 2.3.

The proof of Theorem 2.3 is immediate once we use Theorem 6.2 along with Proposition 2.5. ∎

7 Proofs of Proposition 2.5 and Theorem 2.6

7.1 Tangent Cone Characterization

We fix a $\theta^{*}\in\mathbb{R}^{n\times n}$ and proceed to investigate the tangent cone $T_{{K}^{*}}(\theta^{*}).$ Notice that $K^{*}$ is same as $K(V^{*})$ defined in Section 2.2.1 (see (5.1)). Let $\mathcal{R}^{*}=(R_{1}^{*},R_{2}^{*},\dots,$ $R_{k^{*}}^{*})$ be a partition of $[n]\times[n]$ into $k^{*}$ rectangles where $k^{*}=k(\theta^{*}).$

Recall that the vertices in the grid graph $L_{n}$ correspond to the pairs $(i,j)\in[n]\times[n]$ and its edge set $E_{n}$ consists of:

[TABLE]

For any edge $e\in E_{n}$, we denote by $e^{+}$ and $e^{-}$ the vertices associated with $e$ with respect to the natural partial order. For any $\theta\in\mathbb{R}^{L_{n}}$, we will use $\Delta_{e}\theta$ as a shorthand notation for the (discrete) edge gradient $\theta(e^{+})-\theta(e^{-})$. Thus $\mathrm{TV}(\theta)=\sum_{e\in E_{n}}|\Delta_{e}\theta|$. For a general rectangle $R:=([a_{1},a_{2}]\times[b_{1},b_{2}])\cap\mathbb{Z}^{2}\subset L_{n}$, we define its right boundary as follows:

[TABLE]

While defining the above set, we are using the matrix convention for indexing the vertices of $L_{n}$. Thus, the top-left vertex in the two-dimensional array $L_{n}$ is indexed by $(1,1)$ and the bottom-right vertex by $(n,n)$. Similarly we define the left, top and bottom boundaries of $R$ and denote them by $\partial^{\mathrm{left}}(R)$, $\partial^{\mathrm{top}}(R)$ and $\partial^{\mathrm{bottom}}(R)$ respectively. The boundary of $R$, denoted by $\partial R$, is defined as

[TABLE]

7.1.1 Starting from the definition

The tangent cone $T_{{K}(V^{*})}(\theta^{*})$ is the smallest closed, convex cone containing all the elements $\theta$ in $\mathbb{R}^{n\times n}$ such that $\theta^{*}+\theta\in{K}(V^{*})$ for $V^{*}=\mathrm{TV}(\theta^{*})$. Let $A^{*}\coloneqq\{e\in E_{n}:|\Delta_{e}\theta^{*}|>0\}$ and $(A^{*})^{c}=E_{n}\setminus A.$ Observe that $|\Delta_{e}(\theta^{*}+\theta)|-|\Delta_{e}(\theta^{*})|=|\Delta_{e}\theta|-0=|\Delta_{e}\theta|$ for every edge $e$ in $(A^{*})^{c}$. Thus in order for $\theta^{*}+\theta\in{K}(V^{*})$, the increments in the absolute edge gradients of $\theta^{*}+\theta$ from the edges in $(A^{*})^{c}$ must be compensated by an equal or greater amount of decrease in the absolute edge gradients for the edges in $A^{*}$. The precise statement is the content of

Lemma 7.1**.**

We have the following set equality:

[TABLE]

Here, $sgn(x)\coloneqq\mathbb{I}\{x>0\}-\mathbb{I}\{x<0\}$ is the usual sign function.

Proof.

Let $T$ be the set on the right side of (7.1). Let us first prove that $T_{{K}(V^{*})}(\theta^{*})\subset T$. An important feature of $T$ is that it is a closed convex cone. Hence it suffices to show that $\theta\in T$ whenever $\theta^{*}+\theta\in{K}(V^{*})$. To this end let $\theta$ be such that $\mathrm{TV}(\theta^{*}+\theta)\leq\mathrm{TV}(\theta^{*})$. Since ${K}(V^{*})$ is a convex set, we have

[TABLE]

for any $0\leq c\leq 1$. Now observing that

[TABLE]

we can write

[TABLE]

whenever $c$ is small enough satisfying $sgn(\Delta_{e}\theta^{*}+c\Delta_{e}\theta)=sgn(\Delta_{e}\theta^{*})$ for all $e\in A^{*}$. By definition,

[TABLE]

which together with (7.2) gives us $\theta\in T$.

It remains to show that $T\subset T_{{K}(V^{*})}(\theta^{*}).$ It suffices to show that for any $\theta\in T$ there exists a small enough $c>0$ such that $\mathrm{TV}(\theta^{*}+c\theta)\leq V^{*}$. This can be shown using the same reasoning given after (7.2). ∎

With the above characterization of the tangent cone, we are now ready to prove our lower bound to the risk given in Theorem 2.3.

7.2 Proof of Proposition 2.5

Recall that here we consider $\theta^{*}$ which is piecewise constant on two rectangles and is of the following form:

[TABLE]

Proof.

Consider $n$ to be even and a perfect square (i.e., $\sqrt{n}$ is an integer) for simplicity of exposition. Also for a generic $n\times n$ matrix $\theta$ we will denote $\theta^{(1)}$ to be the submatrix formed by the first $n/2-1$ columns, $v^{(\theta)}$ to be the $n/2$-th column and $\theta^{(2)}$ to be the submatrix formed by the last $n/2$ columns. Also, for two matrices $\theta$ and $\theta^{{}^{\prime}}$ with the same number of rows, we will denote $[\theta:\theta^{{}^{\prime}}]$ to be the matrix obtained by concatenating the columns of $\theta$ and $\theta^{{}^{\prime}}$.

We can now use Lemma 7.1 to characterize the tangent cone $T_{{K}(V^{*})}(\theta^{*})$.

[TABLE]

In this proof, we will actually lower bound the Gaussian width of a convenient subset of $T_{{K}(V^{*})}(\theta^{*})$. To this end, for constants $c_{1},c_{2}\in(0,1)$ to be specified later, let us define

[TABLE]

In words, for $\theta\in S$, the first $n/2-1$ columns are all equal to $c_{1}/n$, the last $n/2$ columns of $\theta$ are [math] and the entries in the $n/2$-th column can take two values; either $c_{2}/\sqrt{n}$ or $c_{1}/n$. Also, for any such matrix $\theta$,

[TABLE]

Before going further, let us define the set of indices $B_{j}\coloneqq\{(j-1)\sqrt{n}+1,\,(j-1)\sqrt{n}+2,\dots,j\>\sqrt{n}\}$ for $j\in[\sqrt{n}].$ In words, we divide $[n]$ into $\sqrt{n}$ many equal contiguous blocks and $B_{j}$ refers to the $j$th block. Now, for any realization of a random Gaussian matrix $Z$, let us define the matrix $\nu$ so that $\nu^{(1)}\coloneqq\frac{c_{1}}{n}\textbf{1}_{n\times(n/2-1)}$ and $\nu^{(2)}\coloneqq\mathbf{0}_{n\times n/2}$. Moreover, we define $v^{(\nu)}$ as follows:

[TABLE]

In words, the vector $v^{(\nu)}$ is defined so that it is constant on each of the blocks $B_{j}$. If $\sum_{i\in B_{j}}Z[i,n/2]>0$, the value on $B_{j}$ is $\frac{c_{2}}{\sqrt{n}}$, otherwise the value is $\frac{c_{1}}{n}$. Now we claim that the following are true for some appropriate choice of $c_{1}$ and $c_{2}$:

a) $\nu\in S$ for any $Z.$

b) $\|\nu\|\leq 1.$

Taking the above claims to be true we can write

[TABLE]

where we used the fact that $\nu^{(1)},\nu^{(2)}$ are constant matrices and $Z$ has mean zero entries.

Now let us denote $\mathcal{Z}_{j}\coloneqq\sum_{i\in B_{j}}Z[i,n/2]$. Note that $(\mathcal{Z}_{1},\dots,\mathcal{Z}_{\sqrt{n}})$ are independent mean zero Gaussians with standard deviation $n^{1/4}$. Therefore

[TABLE]

where for a standard Gaussian random variable $z$, we denote $\phi=\mathbb{E}\,z\mathbb{I}\{z>0\}.$

It remains to choose $c_{1},c_{2}$ so that the two claims hold as well as $c_{2}-\tfrac{c_{1}}{\sqrt{n}}$ is positive. To this end notice that for validating the first claim it suffices to show, in view of the definition of $\nu$, that the first inequality in (7.3) holds for $\nu$, i.e., the following is true

[TABLE]

Now entries of $v^{\nu}$ can take two values, either $\frac{c_{2}}{\sqrt{n}}$ or $\frac{c_{1}}{n}.$ In either case it can be checked that when $c_{2}\geq\tfrac{c_{1}}{\sqrt{n}}$ we have for each row index $i\in[n]$

[TABLE]

Along with the fact that

[TABLE]

(7.5) implies that in order to verify (7.4) it suffices to show $\mathrm{TV}(v^{\nu})\leq c_{1}.$ But $v^{\nu}$ is a piecewise constant vector with at most $\sqrt{n}$ jumps of size $\frac{c_{2}}{\sqrt{n}}$. Thus we have $\mathrm{TV}(v^{\nu})\leq c_{2}$. Hence ensuring $c_{2}\leq c_{1}$ is sufficient to obtain the first claim. The second claim is trivially satisfied if $c_{2}\leq\sqrt{1-c_{1}^{2}}.$ Thus, choosing $c_{1}=c_{2}=1/\sqrt{2}$ we can satisfy both claims as well as $c_{2}-\tfrac{c_{1}}{\sqrt{n}}=\tfrac{1}{\sqrt{2}}(1-1/\sqrt{n})>0$ for all $n\geq 2$. ∎

The task now is to obtain a “matching” upper bound on the gaussian width, which would eventually lead to the proof of Theorem 2.2 in view of Theorem 6.1. Since the proof is lengthy and somewhat technical, for the benefit of the reader we first provide an informal roadmap of the proof before starting it formally.

7.3 Proof of Theorem 2.6

Proof.

Consider the signal matrix $\theta^{*}\coloneqq\mathbb{I}\{i+j>n\}$. From the characterization of the tangent cone given by Lemma 5.3, we have

[TABLE]

where every edge $e$ in $A^{*}$ is either of the form $((i,n-i),(i,n-i+1))$ or $((i,n-i),(i+1,n-i))$ for some $i\in[n-1]$.

Now consider the family $T^{*}$ of matrices defined below:

[TABLE]

It is not difficult to check that $T^{*}\subseteq T_{K(V^{*})}(\theta^{*})$. It is also clear that $T^{*}$ is (linearly) isomorphic to $\mathbb{R}_{+}^{n-1}$. Therefore

[TABLE]

where $B_{m}(r)$ denotes the usual Euclidean ball of radius $r$ in $\mathbb{R}^{m}$ and $c>0$ is a universal constant. Now an application of Theorem 6.2 along with the above Gaussian width lower bound also furnishes a lower bound to the limiting MSE. ∎

Remark 7.1**.**

The vertex boundary of a set $A\subset L_{n}$ is defined to be the set of vertices which share an edge with $A^{c}$. Consider the level sets of $\theta^{*}$ which are the sets $A=\{(i,j)\in L_{n}:i+j>n\}$ and $A^{c}.$ The simple argument presented in the proof of Theorem 2.6 relies crucially on the fact that the vertex boundary of the level sets $A$ and $A^{c}$ are not connected in the graph $L_{n}$. One can now consider other signals of the form $\theta^{*}=\mathbb{I}\{A\}$ for a general subset $A\subset L_{n}$. One can check that if $A$ is of the shape of a circle or a square rotated by $45$ degrees then also the vertex boundary of the level sets will contain $O(n)$ connected components which are singletons. Therefore, a similar argument will give a $O(n)$ lower bound to $\mathcal{GW}(T_{K(V^{*})}(\theta^{*}).$ We believe that it might be possible to formalize the intuition that whenever $A$ is sufficiently far from being a rectangle, $\mathcal{GW}(T_{K(V^{*})}(\theta^{*})$ is lower bounded by $O(n).$

8 Proof of Proposition 2.4

8.1 Informal Roadmap

The proof of Proposition 2.4 can be divided into three major steps which we now describe. Recall that $\theta^{*}$ is the true signal which is piecewise constant on axis aligned rectangles $R_{1}^{*},\dots,R_{k(\theta^{*})}^{*}$ which partition $L_{n}.$

Step 1: We have to bound $\mathcal{GW}(T_{{K}(V^{*})}(\theta^{*})\cap B_{n\times n}(1)).$ To do this, we show that if a matrix $\theta$ is in $T_{{K}(V^{*})}(\theta^{*})\cap B_{n\times n}(1)$ then each rectangular submatrix $\theta_{R_{i}^{*}}$ satisfies the property that $\mathrm{TV}(\theta_{R_{i}^{*}})$ is at most the $\ell_{1}$ norm of its four boundaries plus a small wiggle room $\delta>0.$ Such matrices are denoted later in (8.2) as $\mathcal{M}^{4}.$ This fact then reduces our problem to bounding the Gaussian width for the class of matrices $\mathcal{M}^{4}.$ Corollary 8.1, Lemma 8.2 and Lemma 8.3 are part of this step.

Step 2: Before starting the Gaussian width calculations, we found it convenient to further simplify the class of matrices $\mathcal{M}^{4}.$ In this step, we show that if a matrix $\theta$ lies in $\mathcal{M}^{4}$ then we can subdivide it further into several submatrices which now satisfy a simpler property. The property is that the total variation of these submatrices are at most the $\ell_{1}$ norm of only one or none of its boundaries (instead of four) plus an appropriately small “wiggle room” $\delta>0.$ These sets of matrices are denoted by $\mathcal{M}^{1}$ and $\mathcal{M}^{0}$ respectively and are defined just before Lemma 8.4. Along with Lemma 8.4, Lemmata 8.5–8.7 are also parts of this step.

Step 3: This is the step where we actually compute the metric entropies of the classes of matrices $\mathcal{M}^{1}$ and $\mathcal{M}^{0}$ and finally bring all the pieces together. Proposition 8.9 and Lemmata 8.8–8.14 are all parts of this step.

8.2 Towards simplifying the tangent cone

We first want to split $\theta$ into submatrices each of which satisfies a separate constraint. This and the next subsection are devoted to this goal. Let us revisit Lemma 7.1. Since $\theta^{*}$ is constant on each rectangle $R_{i}^{*}\in\mathcal{R}^{*}$ it follows that

[TABLE]

As a consequence we get the following corollary:

Corollary 8.1**.**

Fix $\theta^{*}\in\mathbb{R}^{n\times n}$. We have

[TABLE]

The first step towards obtaining a decomposition where each submatrix satisfies some constraint is to separate the constraints for $R_{i}^{*}$’s. More precisely we would like

[TABLE]

for each $i\in[k^{*}]$. As we will see below that this is “almost” the truth when we consider matrices in the tangent cone which are of unit norm.

Let us make precise the notion of an “almost” version of (8.1). To this end we introduce for any $\delta,\,t>0$:

[TABLE]

where $\theta_{\mathrm{left}}\coloneqq\theta[\>,\>1]$, $\theta_{\mathrm{right}}\coloneqq\theta[\>,\>n^{\prime}]$, $\theta_{\mathrm{top}}\coloneqq\theta[1,\>]$ and $\theta_{\mathrm{bottom}}\coloneqq\theta[m^{\prime},\>]$. In plain words, $\mathcal{M}^{4}(m^{\prime},n^{\prime},\delta,t)$ consists of matrices of norm at most $t$ whose total variation is bounded by the total $\ell_{1}$ norm of its four boundaries plus an extra wiggle room $\delta>0$. In our next result we show that for any $\theta$ in $T_{K(V^{*})}(\theta^{*})$ intersected with the unit Euclidean ball $B_{n\times n}(1)$, the restriction $\theta_{|R_{i}^{*}}$ of $\theta$ to $R_{i}^{*}$ lies in $\mathcal{M}^{4}(m_{i},n_{i},\delta_{i},t_{i})$ for each $i\in[k]$ with $m_{i}:=\mathrm{n}_{\mathrm{row}}(R_{i}^{*})$, $n_{i}:=\mathrm{n}_{\mathrm{col}}(R_{i}^{*})$ and $t_{i}$’s and $\delta_{i}$’s satisfying some upper bounds on their $\ell_{2}$ and $\ell_{1}$ norms respectively.

Lemma 8.2**.**

*We have the set inclusion *

[TABLE]

where $S_{k^{*},r}:=\{a\in\mathbb{R}^{k^{*}}_{+}:\sum_{i\in[k^{*}]}a_{i}\leq r\}$ is the non negative simplex with radius $r>0$, $\bm{t^{2}}$ is the vector $(t_{1}^{2},\dots,t_{k^{*}}^{2})\in\mathbb{R}_{+}^{k^{*}}$ and

[TABLE]

Remark 8.1**.**

By virtue of Lemma 8.2, we achieve our objective of obtaining a characterization of $T_{K(V^{*})}(\theta^{*})$ where we have separate constraints for each $R_{i}^{*}\in\mathcal{R}^{*}.$ The constraints are now coupled together by the wiggle room vector $\bm{\delta}\in S_{k^{*},\Delta(\theta^{*})}$ and the (squared) $\ell_{2}$ norm vector $\bm{t^{2}}$.

Proof.

We will start with a claim.

Claim 8.1**.**

Let $\theta\in T_{K(V^{*})}(\theta^{*})\,\cap\,B_{n\times n}(1).$ Then for each $i\in[k^{*}]$ and any fixed choice of rows and columns $r_{i},\,c_{i}$ in $R_{i}^{*}$, we have $\theta_{|R_{i}^{*}}\in\mathcal{M}^{4}(m_{i},n_{i},\delta_{i},t_{i})$ where $\sum_{i\in[k^{*}]}t_{i}^{2}\leq 1$ and $(\delta_{1},\dots,\delta_{k^{*}})=:\bm{\delta}\in\mathbb{R}^{k^{*}}_{+}$ satisfies

[TABLE]

where $\theta_{|c_{i}}$ (or $\theta_{|r_{i}}$) is the vector obtained by restricting $\theta$ to the row $c_{i}$ (respectively the column $r_{i}$).

Let us first deduce the lemma assuming our claim. Consider a $\theta\in T_{K(V^{*})}(\theta^{*})$ such that $\|\theta\|_{2}\leq 1$ and for each $i\in[k^{*}]$, let $r_{i}$ and $c_{i}$ denote the rows and columns such that the $\ell_{1}$ norms of $\theta_{|r_{i}}$ and $\theta_{|c_{i}}$ are minimum. Then by Claim 8.1, each $\theta_{|R_{i}^{*}}\in\mathcal{M}^{4}(m_{i},n_{i},\delta_{i},t_{i})$ with $\bm{t^{2}}\in S_{k^{*},1}$ and $\bm{\delta}$ satisfying

[TABLE]

Now for each $i\in[k^{*}]$, we have

[TABLE]

The first inequality is an application of the Cauchy-Schwarz inequality and the second inequality follows from the “minimum is less than the average” principle. Similarly, one can obtain the row version of these inequalities and together they give us

[TABLE]

Summing the above inequality over all $i\in[k^{*}]$ and subsequently using the Cauchy-Schwarz inequality as well as the fact that $\|\theta\|_{2}\leq 1$, we get in view of (2.2)

[TABLE]

thus yielding the lemma.

Proof of Claim 8.1. The constraint on $t_{i}$’s is clear and therefore all we need to show is the constraint on $\delta_{i}$’s. Recall from the definition in (8.2) that $\delta_{i}$ can be chosen, for any $i\in[k^{*}]$, as

[TABLE]

where $a_{+}\coloneqq\max\{a,0\}$ for any $a\in\mathbb{R}$. Now fix $i\in[k^{*}]$ and consider a generic row $r_{i}$ of $R_{i}^{*}$. Treating $r_{i}$ as a horizontal path in the graph $L_{n}$, let us denote its two end-vertices by $u$ and $w$ with $u\in\partial^{\mathrm{left}}(R_{i}^{*})$ and $w\in\partial^{\mathrm{right}}(R_{i}^{*})$. Now denoting the vertex in $r_{i}\,\cap\,c_{i}$ by $v$, we see that $v$ occurs between the vertices $u$ and $w$ in the row $r_{i}$. Therefore we can write

[TABLE]

Summing the above inequality for every row in the rectangle $R_{i}^{*}$ gives us

[TABLE]

By a similar argument applied to the columns of $R$ we obtain

[TABLE]

Summing the previous two displays we get the following inequality:

[TABLE]

Now if $\theta\in T_{K(V^{*})}(\theta^{*})$, then as a consequence of Corollary 8.1 we also have

[TABLE]

Hence an application of Lemma 10.1 (stated and proved in the appendix) to $f_{i}=\mathrm{TV}(\theta_{|R_{i}^{*}})$, $g_{i}=\sum_{u\in\partial R_{i}^{*}}|\theta(u)|$, $h_{i}=2(\left\lVert\theta_{r_{i}}\right\rVert_{1}+\left\lVert\theta_{c_{i}}\right\rVert_{1})$ and $w_{i}=\delta=0$, would give us the claim in view of (8.4). ∎

With the help of Lemma 8.2 we can now deduce the following lemma.

Lemma 8.3**.**

With the notation described in this section, we have the following upper bound:

[TABLE]

where $H_{k^{*}}\coloneqq\{\frac{1}{k^{*}},\frac{2}{k^{*}},\dots,1\}^{k^{*}}$ and $C>0$ is a universal constant.

Proof.

Using Lemma 8.2 we can write

[TABLE]

where, by a slight abuse of notation, $Z$ always refers to a matrix of independent standard normals with appropriate number of rows and columns.

At this point, we would like to convert the supremum over $\bm{\delta},\bm{t^{2}}$ (or, equivalently $\bm{t}$) in the non negative simplex to a maximum over a finite net of $\bm{\delta},\bm{t}$ We can accomplish this by the following trick. Fix any $\bm{\delta}\in S_{k^{*},\Delta(\theta^{*})}.$ Then we can define a vector $\bm{q}=\bm{q}(\bm{\delta})\in\mathbb{R}^{k^{*}}$ such that

[TABLE]

It is clear that $\mathbf{q}\in H_{k^{*}}\cap S_{k^{*},2}$. It is also clear that $\bm{\delta}\leq\bm{q}\>\Delta(\theta^{*})$ element-wise. Due to similar reason, for any $\bm{t^{2}}\in S_{k^{*},1}$ there exists $\bm{w}=\bm{w}(\bm{t})\in H_{k^{*}}\cap S_{k^{*},2}$ such that $\bm{t^{2}}\leq\bm{w}$ element-wise. Since the collections $\mathcal{M}^{4}(m_{i},n_{i},\delta_{i},t_{i})$ are increasing in $(\delta_{i},t_{i})$ (with respect to set inclusion), it follows from the previous discussion that

[TABLE]

Since $Z$ is a matrix with i.i.d $N(0,1)$ entries, the first two maximums in the right hand side of the above display can actually be taken outside the expectation upto an additive term. This follows from the well known concentration properties of suprema of gaussian random variables. In particular, we now apply Lemma 10.2 (stated in the appendix), true for suprema of gaussians, to obtain for a universal constant $C,$

[TABLE]

To bound the log cardinality $\log|H_{k^{*}}\cap S_{k^{*},2}|$, note that for any positive integer $k^{*}$, the cardinality $|H_{k^{*}}\cap S_{k^{*},2}|$ is the same as the number of $k^{*}$ tuples of positive integers summing up to at most $2k^{*}$. By standard combinatorics, we have

[TABLE]

Since

[TABLE]

for all $s\in\{k^{*},\ldots,2k^{*}-1\}$, it follows that

[TABLE]

for some positive absolute constant $C$.

Using (8.2), (8.2), (8.2) and the above cardinality bound, we can finally finish the proof by writing

[TABLE]

Operationally, the above lemma reduces the task of upper bounding the Gaussian width of $T_{K(V^{*})(\theta^{*})}\cap B_{n,n}(1)$ to upper bounding the Gaussian width of $\mathcal{M}^{4}$ with appropriate parameters. However, it would be convenient for us to bound the Gaussian width when the number of boundaries involved in the constraint is at most one instead of four. The results in the next subsection makes this possible.

8.3 Further simplification: from four boundaries to one

We now proceed to the second step, i.e., reducing the number of boundaries involved in the constraints from four to one (or zero). Thus, we will keep on subdividing each $\theta_{|R_{i}}$ until we obtain submatrices satisfying constraints similar to (8.2), albeit with the $\ell_{1}$-norm of at most one boundary vector appearing on the right hand side of the bound on total variation. This is the content of this subsection.

Taking the cue from the the previous subsection, let us define

[TABLE]

We can define $\mathcal{M}^{\mathrm{bottom}}(m^{\prime},n^{\prime},\delta,t)$, $\mathcal{M}^{\mathrm{left}}(m^{\prime},n^{\prime},\delta,t)$ and $\mathcal{M}^{\mathrm{right}}(m^{\prime},n^{\prime},\delta,t)$ in a similar fashion. Notice that the constraint satisfied by the total variation of the members of $\mathcal{M}^{\mathrm{right}}(m^{\prime},n^{\prime},\delta,t)$ is “almost” identical to (7.3). By abuse of notation we will refer to any of the four families of matrices described above by a generic notation which is $\mathcal{M}^{1}(m^{\prime},n^{\prime},$ $\delta,t)$. The reason behind this is that our ultimate concerns would be the Gaussian widths of these families which, for $m^{\prime}$ and $n^{\prime}$ close enough to each other, are expected to be of similar order by symmetry. Using a single notation for them would thus minimize the notational clutter. In a similar vein we define

[TABLE]

Having defined the relevant families of matrices, we can now state our main result for this subsection.

Lemma 8.4**.**

Fix positive integers $m,n$ and positive numbers $\delta,t.$ Define for each integer $j\geq 1$,

[TABLE]

Then we have the following bound for a universal constant $C>0$,

[TABLE]

Here, to simplify notations, we use $m/2^{j}$, for $m,j\in\mathbb{N}$, to denote any (but fixed in any given context) integer $m^{\prime}$ between $m2^{-(j+1)}$ and $m2^{-j}$. The similar definition for $n$ instead of $m$ is denoted by $n/2^{j}.$ $K$ equals the number of binary divisions of $[m]\times[n]$ on both axes that are possible and equals $\min\{\log_{2}m,\log_{2}n\}$ up to a universal constant.

The above lemma bounds the Gaussian width of $\mathcal{M}^{4}$ in terms of Gaussian widths of simpler classes of matrices $\mathcal{M}^{1}$ and $\mathcal{M}^{0}$. We devote the next subsection to its proof.

8.4 Proof of Lemma 8.4

We need some intermediate lemmas. We start with the following lemma. The notation convention is same as in Lemma 8.4.

Lemma 8.5**.**

There exists a rectangular partition $\mathcal{R}$ of $[m]\times[n]$ with the following property. For any $\theta\in\mathcal{M}^{4}(m,n,\delta,t)$, there exists non negative real numbers $t_{R}$ for every rectangle $R\in\mathcal{R}$ such that:

• •

$\mathcal{R}=\bigcup_{j\in[K],\,k\in[2]}\mathcal{R}_{j,k}$* where $\mathcal{R}_{j,k}$’s are disjoint sets of rectangles and all the rectangles in $\mathcal{R}_{j,k}$ are of size $m_{i}/2^{j}\times n_{i}/2^{j}$.*

• •

$|\mathcal{R}_{j,1}|\leq 8$* and for any $R\in\mathcal{R}_{j,1}$ we have $\theta_{|R}\in\mathcal{M}^{1}(m/2^{j},n/2^{j},\delta^{(j)},t_{R})$.*

• •

$|\mathcal{R}_{j,2}|\leq 4$* and for any $R\in\mathcal{R}_{j,2}$ we have $\theta_{|R}\in\mathcal{M}^{0}(m/2^{j},n/2^{j},\delta^{(j)},t_{R})$.*

• •

$\sum_{R\in\mathcal{R}}t_{R}^{2}=t^{2}$**

Proof of Lemma 8.4.

The proof of Lemma 8.4 follows directly from Lemma 8.5 and the sub-additivity of the Gaussian width functional. ∎

The task now is to prove Lemma 8.5. The proof of Lemma 8.5 is divided into two steps where we state and prove two intermediate lemmas. In the first step we reduce the number of “active” boundaries, i.e., the number of boundary vectors involved in the bound on total variation, from four to two and in the second step we reduce them from two to one or zero. The main idea of the proofs is essentially same as that of Lemma 8.2.

Remark 8.2**.**

While lemma 8.5 is true for any integers $m,n$, the reader can safely read on as if $m,n$ are powers of $2$. The essential aspects of the proof of Lemma 8.5 all go through in this case. Writing the general case would make the notations messy. For the sake of clean exposition, we thus write the entire proof when $m$ and $n$ are powers of $2$. At the end, we mention the modifications needed when $m,n$ are not powers of $2$.

**Four to two boundaries. **

In order to state this result let us define for any $\delta>0,$

[TABLE]

Similarly we can define the families $\mathcal{M}^{\mathrm{topleft}}(\cdots),\,\mathcal{M}^{\mathrm{bottomleft}}(\cdots)$ and $\mathcal{M}^{\mathrm{bottomright}}(\cdots)$. Likewise $\mathcal{M}^{1}(m,n,\delta,t)$, we will refer generically to any of these four families of matrices by $\mathcal{M}^{2}(m,n,$ $\delta,t)$. Below we call a partitioning of a matrix $\theta\in\mathbb{R}^{m\times n}$ as an equal dyadic partitioning if each submatrix lies in $\mathbb{R}^{m/2\times n/2}$ and is formed by adjacent rows and columns of $\theta$ as $\theta^{\mathrm{topleft}},\,\theta^{\mathrm{topright}},$ $\theta^{\mathrm{bottomleft}}$ and $\theta^{\mathrm{bottomright}}$ in the obvious order.

Lemma 8.6**.**

Take any $\theta\in\mathcal{M}^{4}(m,n,\delta,t)$. Let us denote the four submatrices obtained by an equal dyadic partitioning of $\theta$. Then the submatrix $\theta^{ab}$, where $a\in\{\mathrm{top},\mathrm{bottom}\}$ and $b\in\{\mathrm{left},\mathrm{right}\}$, itself satisfies

[TABLE]

In words, if a matrix $\theta\in\mathcal{M}^{4}(m,n,\delta,t)$ is dyadically partitioned into four equal sized submatrices, each of these four submatrices lies in $\mathcal{M}^{2}(m/2,n/2,\delta^{\prime},t)$ where $\delta^{\prime}\coloneqq\delta+16t(\sqrt{\frac{{m}}{{n}}}+\sqrt{\frac{{n}}{{m}}})$; furthermore the boundaries that are active for these submatrices are the ones that they share with $\theta$.

Proof.

Since $\left\lVert\theta\right\rVert_{2}\leq t$, there exists $1\leq i\leq m/2<i^{\prime}\leq m$ and $1\leq j\leq n/2<j^{\prime}\leq n$ such that

[TABLE]

The previous display and the Cauchy-Schwarz inequality together imply

[TABLE]

Now consider the submatrix $\theta^{\mathrm{topleft}}$ for which we have

[TABLE]

where in the last step we used the fact that $\theta^{\mathrm{topleft}}[\,,\,j]=\theta[1:m/2,\,j]$. A similar argument gives us

[TABLE]

Analogous lower bounds for ${\mathrm{TV}}_{\mathrm{row}}$ and ${\mathrm{TV}}_{\mathrm{col}}$ of the other three submatrices can be derived involving the $\ell_{1}$ norms of appropriate boundaries and (partial) rows or columns of $\theta$. Adding all these together and using (8.4), we obtain

[TABLE]

On the other hand, since $\theta\in\mathcal{M}^{4}(m,n,\delta,t)$ we have

[TABLE]

An application of Lemma 10.1 now finishes the proof of the lemma from the previous two displays. ∎

**Two to one or zero boundary. **

Let us start by stating the following lemma which one can think of as a version of Lemma 8.6 applied to an element of $\mathcal{M}^{2}(m,n,\delta,t)$. The proof is very similar and we leave it to the reader to verify.

Lemma 8.7**.**

Let $\theta\in\mathcal{M}^{ab}(m,n,\delta,t)$ for some $a\in\{\mathrm{top},\mathrm{bottom}\}$ and $b\in\{\mathrm{left},\mathrm{right}\}.$ We can partition $\theta$ into equal sized four submatrices $\theta^{\mathrm{topleft}},\theta^{\mathrm{topright}},$ $\theta^{\mathrm{bottomleft}}$ and $\theta^{\mathrm{bottomright}}$ in the obvious manner such that the submatrix $\theta^{cd}$, where $c\in\{\mathrm{top},$ $\mathrm{bottom}\}$ and $d\in\{\mathrm{left},\mathrm{right}\}$, satisfies

[TABLE]

In words, if a matrix $\theta\in\mathcal{M}^{2}(m,n,\delta,t)$ is dyadically partitioned into four equal sized submatrices, then each of these four submatrices has at most two active boundaries and a wiggle room of at most $\delta+16t(\sqrt{\frac{{m}}{{n}}}+\sqrt{\frac{{n}}{{m}}})$; furthermore the active boundaries are the ones that they share with one of the active boundaries of $\theta$.

We are now ready to conclude the proof of Lemma 8.5.

Proof of Lemma 8.5.

Recall that we are assuming $m,n$ are powers of $2$ for simplicity of exposition.

Step 0: Partition $[m]\times[n]$ dyadically into four equal rectangles so that for any such rectangle $S$, $\theta_{|S}\in\mathcal{M}^{2}(m/2,n/2,\delta^{(0)},\left\lVert\theta_{|S}\right\rVert)$ by Lemma 8.6 where

[TABLE]

Step 1: Let $S$ (there are four of them) be a generic rectangle obtained from the previous step. Using Lemma 8.7, we now partition $\theta_{|S}$ into four equal parts (rectangles). We then get two matrices in $\mathcal{M}^{1}(m/4,n/4,\delta^{(1)},t)$, one matrix in $\mathcal{M}^{0}(m/4,n/4,\delta^{(1)},t)$ and the remaining one from $\mathcal{M}^{2}(m/4,n/4,\delta^{(1)},t)$. Here,

[TABLE]

Steps $\bm{j}\mathbf{\geq}\mathbf{2}$: From the last step we get exactly one matrix in $\mathcal{M}^{2}(m/4,n/4,\delta^{(1)},t)$, for each of the $4$ rectangles $S.$ For each $S,$ we now recursively use Lemma 8.7 by partitioning this matrix again into four exactly equal parts in a dyadic fashion and continue the same procedure with the matrix obtained in each step with two active boundaries until we end up with matrices only with 0 or 1 active boundary. Observe that in the very last step we arrive at a submatrix with exactly one row or column in place of the one with two active boundaries.

For each $j\geq 1$, define $\mathcal{R}_{j,1}$ as the collection of rectangles $R$ obtained in step $j$ such that $\theta_{|R}$ has exactly $1$ active boundary. From Lemma 8.7, we know that there are exactly two such rectangles for any given $S$ (from step [math]) and therefore $|\mathcal{R}_{j,1}|\leq 8$. For any $j\geq 1,$ and any rectangle $R\in\mathcal{R}_{j,1}$, repeated application of Lemma 8.7 yields that $\theta_{|R}\in\mathcal{M}^{1}(m/2^{j},n/2^{j},\delta^{(j)},\left\lVert\theta_{|R}\right\rVert)$ where

[TABLE]

Now defining $\mathcal{R}_{j,2}$ as the collection of rectangles $R$ obtained in step $j$ such that $\theta_{|R}$ has no active boundary, we can deduce in a similar way that $|\mathcal{R}_{j,2}|\leq 4.$ Also for such rectangles $R$ and $j\geq 1$ we have $\theta_{|R}\in\mathcal{M}^{0}(m/2^{j},n/2^{j},\delta^{(j)},\left\lVert\theta_{|R}\right\rVert)$. Finally, notice that

[TABLE]

Thus the collection of rectangles $\{\mathcal{R}_{j,k}:j\geq 1,k\in[2]\}$ satisfies all the conditions of Lemma 8.5.

∎

Remark 8.3**.**

For the statement of Lemma 8.4 to hold, the important thing in the proof of Lemma 8.5 is that in every step $1\leq j\leq K$, the aspect ratio of the submatrices does not change significantly. The reader can check that at every step, both the number of rows and columns halve, thus keeping the aspect ratio constant. At every step, the dimensions of the submatrices halve and thus decrease geometrically, while the allowable wiggle room increases additively by the factor (does not change with $j$) $16\>t(\sqrt{\frac{m}{n}}+\sqrt{\frac{n}{m}})$.

Remark 8.4**.**

Let us discuss the case when $m,n$ are not necessarily powers of $2$ in the proof of Lemma 8.5. The first step of reducing the number of active boundaries from four to two, by applying Lemma 8.6, can be carried out in the same way by splitting at the point $\lfloor m/2\rfloor$ and $\lfloor n/2\rfloor$. Next, we come to the stage when we are applying Lemma 8.7 to reduce the number of active boundaries from two to one, on the four submatrices obtained from the previous step. Let us denote the dimensions of these $4$ submatrices generically by $m^{\prime},n^{\prime}$. Recall, in the first step of subdivision, we get exactly one submatrix with $2$ active boundaries. The others have $1$ or [math] active boundaries. At this step, we can subdivide such that the submatrix with two active boundaries has dimensions which are exactly powers of $2$. For instance, we can split at the unique power of $2$ between $m^{\prime}/4$ and $m^{\prime}/2$ on one dimension and do the exact same thing for the other dimension. Once we have this submatrix with two active boundaries to have dimensions which are exactly powers of $2$, we can carry out the rest of the steps as in the proof of Lemma 8.5. It can be checked that, in this case, all the inequalities we deduce while proving Lemma 8.4 goes through with the possible mutiplication of a universal constant.

8.5 Upper bounds on Gaussian Widths and the proof of Proposition 2.4

Now that we have reduced the problem of bounding the gaussian width of $T_{K(V^{*})(\theta^{*})}\cap B_{n,n}(1)$ to that of $\mathcal{M}^{0}(m,n,\delta,t)$ and $\mathcal{M}^{1}(m,n,\delta,t)$, we need to obtain upper bounds on these quantities in order to conclude the proof of Theorem 2.2. Our next lemma provides an upper bound on the gaussian width of $\mathcal{M}^{0}(m,n,\delta,t)$ which we henceforth denote as $\mathcal{GW}^{0}(m,n,\delta,t)$.

Lemma 8.8**.**

Fix $\delta>0$ and $t\in(0,1]$. For positive integers $m$ and $n$ such that $\max\{m/n,n/m\}\leq c$ for some $c>0$, we have the following upper bound on the Gaussian width:

[TABLE]

where $C$ is a constant depending only on $c$.

Proof.

Since $m$ and $n$ are of the same order, the bound computed in (5.2) from Section 5 remains valid in this case. ∎

In our next proposition, we provide an upper bound on $\mathcal{GW}^{1}(m,n,\delta,t)$, i.e., the gaussian width of $\mathcal{M}^{1}(m,n,\delta,t)$. This is the main result in this subsection and one of the main technical contributions of this paper.

Proposition 8.9**.**

Fix $\delta\in(0,n]$ and $t\in(0,1]$. Then for positive integers $m,n$ satisfying the conditions of the previous lemma, we have the following upper bound on the Gaussian width:

[TABLE]

Here $x^{2\downarrow}\coloneqq x+x^{2}$ and $C>0$ is a constant depending solely on $c$.

We will prove the above proposition slightly later. Lemma 8.8 and Proposition 8.9 together with Lemma 8.4 now imply (with $\mathcal{GW}^{4}(m,n,\delta,t)$ denoting the gaussian width of $\mathcal{M}^{4}(m,n,\delta,t)$)

Lemma 8.10**.**

Under the same condition as in the previous proposition, we have

[TABLE]

where $C>0$ is a universal constant.

The proof just involves collecting all the relevant terms and adding them up. The reader can safely skip the proof in the first reading.

Proof.

In this proof, we write $a\lesssim b$ to mean $a\leq C\,b$ for some positive constant $C$ — depending at most on the aspect ratio $c$ — whose exact value can change from line to line. Recall that Lemma 8.4 implies for $K\lesssim\log n$,

[TABLE]

First we compute, in view of Proposition 8.9,

[TABLE]

where we have repeatedly used $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ and in the last inequality we have summed up the geometric series. On the other hand, Lemma 8.8 implies

[TABLE]

We can now deduce the lemma from the last two displays. ∎

With the help of the above lemma we can now conclude the proof of Proposition 2.4.

Proof of Proposition 2.4.

Throughout this proof we will use the notation $C$ to denote some positive constant — depending at most on the aspect ratio $c$ like in the previous proof — whose exact value may change from one line to the next. Also we will use “$a\lesssim b$” to mean “$a\lesssim Cb$”. Recall that by Lemma 8.3, $\mathcal{GW}(T_{K(V^{*})}(\theta^{*})\cap B_{n,n}(1))$ is at most

[TABLE]

Now we plug in the bound from Lemma 8.10 to obtain a bound on the sum inside the two maximums in the above display:

[TABLE]

Since the aspect ratios of each of the rectangular level sets of $\theta^{*}$ are bounded by a constant, we have $\sum_{i=1}^{k^{*}}{n_{i}}^{2}\lesssim n^{2}$. This can be seen as follows:

[TABLE]

Therefore, we can repeatedly apply the Cauchy-Schwarz inequality to deduce for $\delta,\,\bm{t^{2}}\in S_{k^{*},2}$,

[TABLE]

Also because of constant aspect ratio, we have

[TABLE]

Combining the last two displays we notice that $(\log n)^{9/2}(k^{*})^{5/8}n^{1/4}$ emerges as the dominant term and hence

[TABLE]

Together with (8.11) this finishes the proof. ∎

All that remains towards the proof of Proposition 2.4 is Proposition 8.9. The proof of this proposition is fairly involved. The rest of this section is devoted to its proof.

8.6 Proof of Proposition 8.9

By symmetry, it is enough to bound $\mathcal{GW}(\mathcal{M}^{\mathrm{right}}(m,n,\delta,t))$. To this end, let us introduce a new class of matrices as follows:

[TABLE]

where, let us recall, that the total variation ${\mathrm{TV}}_{\mathrm{row}}(\theta)$ along rows is defined as

[TABLE]

and ${\mathrm{TV}}_{\mathrm{col}}(\theta)\coloneqq{\mathrm{TV}}_{\mathrm{row}}(\theta^{T})$.

The following lemma gives an upper bound of $\mathcal{GW}(\mathcal{M}^{\mathrm{right}})$ in terms of the Gaussian widths of $\mathcal{A}$ with appropriate parameters.

Lemma 8.11**.**

Let $k$ denote the smallest integer satisfying $(1+2+\dots 2^{k})\geq n$. Then we have the following inequality:

[TABLE]

where $n_{j}=2^{j}$ for $j\in[k-1]$ and $n=\sum_{j\in[k]}n_{j}$.

Proof.

The proof proceeds by dividing the $n$ columns into blocks of geometrically increasing length and showing that for any $\theta\in\mathcal{M}^{\mathrm{right}}(m,n,\delta,t)$ the submatrices defined by the blocks live in $\mathcal{A}$ with appropriate parameters. Let $\theta\in\mathcal{M}^{\mathrm{right}}(m,n,\delta,t)$ and subdivide $\theta$ into submatrices $[\theta^{(k)}\rvert\theta^{(k-1)}\rvert\cdots\rvert\theta^{(1)}]$ where $\theta^{(j)}$ has $n_{j}$ many columns. Therefore it suffices to prove that

[TABLE]

for all $j\in[k]$ as $\sqrt{n_{j}/n_{j-1}}<2$. Since $\left\lVert\theta^{(j)}\right\rVert\leq\left\lVert\theta\right\rVert\leq t$, we only need to verify the required bounds on ${\mathrm{TV}}_{\mathrm{col}}(\theta^{(j)})$ and ${\mathrm{TV}}_{\mathrm{row}}(\theta^{(j)})$.

Verifying the bound on ${\mathrm{TV}}_{\mathrm{col}}(\theta^{(j)})$ . We will prove the stronger statement ${\mathrm{TV}}_{\mathrm{col}}(\theta)\leq t\sqrt{m/n}+\delta$. Since $\left\lVert\theta\right\rVert\leq t$ and $\left\lVert\theta\right\rVert^{2}=\sum_{\ell\in[n]}\left\lVert\theta[\,,\>\ell]\right\rVert^{2}$, it follows that $\left\lVert\theta[\,,\>\ell^{*}]\right\rVert\leq t/\sqrt{n}$ for some $\ell^{*}\in[n]$ and hence $\left\lVert\theta[\,,\>\ell^{*}]\right\rVert_{1}\leq t\sqrt{m/n}$ by the Cauchy-Schwartz inequality. Now using the condition that $\mathrm{TV}(\theta)\leq\left\lVert\theta[\,,\>n]\right\rVert_{1}+\delta$ (from the definition of $\mathcal{M}^{\mathrm{right}}(m,n,\delta,t)$), we get

[TABLE]

Thus ${\mathrm{TV}}_{\mathrm{col}}(\theta)\leq\delta+t\sqrt{m/n}$.

Verifying the bound on ${\mathrm{TV}}_{\mathrm{row}}(\theta^{(j)})$ . Let us start with $\theta^{(1)}$. By the Cauchy-Schwartz inequality, $\left\lVert\theta[\,,\>n]\right\rVert_{1}\leq\sqrt{m}\left\lVert\theta[\,,\>n]\right\rVert_{2}\leq t\sqrt{m}$ and thus

[TABLE]

Next consider $\theta^{(j)}$ for some $j\geq 2$. Since $\left\lVert\theta^{(j-1)}\right\rVert_{2}^{2}\leq t$ and it has $n_{j-1}$ columns, there is a column of $\theta^{(j-1)}$ whose $\ell_{2}$-norm is at most $t/\sqrt{n_{j-1}}$. Suppose this column is $\theta[\,,\>a]$. Then a calculation similar to (8.6) yields,

[TABLE]

But this implies, along with the Cauchy-Schwartz inequality, that

[TABLE]

It therefore suffices, in view of the previous lemma, to bound the gaussian width of each $\mathcal{A}(m,n_{j},$ $2t\sqrt{m/n_{j}}+\>\delta,\>t\sqrt{m/n}+\delta,t)$ from above in order to bound $\mathcal{GW}^{\mathrm{right}}(m,n,\delta,t)$. Defining $a=\sqrt{m/n_{j}}$, we can write

[TABLE]

Notice that we suppressed the dependence on $m,n,\delta$ and $t$ which henceforth refer to the corresponding parameters in Proposition 8.9.

In our next result, which is crucial for the proof of Proposition 8.9, we give a subspace cover for the set $\mathcal{A}_{a}$ corresponding to any distance $\tau$ between $1/m$ and 1.

Lemma 8.12**.**

Let $t\leq 1$, $\tau\in[1/m,1]$ and $a\geq c$ be such that $m/a^{2}$ is a positive integer between $1$ and $n$. Here $c$ is from the statement of Proposition 8.9. Then there exists a $\tau$ subspace cover $\mathcal{S}_{\tau}$ of $\mathcal{A}$, depending on $m,n,a,\delta$ and $t$ in addition to $\tau$, and a constant $C>0$ depending solely on $c$ such that

[TABLE]

where $\mathcal{L}(x)\coloneqq x\log(\mathrm{e}\log(\mathrm{e}m)^{2}x)$ and

[TABLE]

*(recall that $x^{2\downarrow}\coloneqq x+x^{2}$). *

Remark 8.5**.**

Notice that $\mathcal{L}(x)$ is linear in $x$ ignoring the $\log$ factors. Thus it is helpful to read the above bound as scaling like $\tfrac{\sqrt{m}}{\tau^{2}}$ up to log factors and the lower order terms. This $\sqrt{m}$-scaling is crucial for us in order to derive the $1/4$ exponent of $n$ in Proposition 8.9 and subsequently the correct exponent of $n$ in Theorem 2.2.

Remark 8.6**.**

The reason for assuming a polynomial lower bound (in $m$) on $\tau$ is that we want $\log(1/\tau)$ to be at most $O(\log m)$. Hence the bounds of Lemma 8.12 remain valid, with appropriate changes in $C$, as long as $\tau\geq 1/m^{c}$ for some universal constant $c>0$.

With Lemma 8.12 we can now finish the proof of Proposition 8.9.

Proof of Proposition 8.9.

An important feature of the bounds in Lemma 8.12 is that it does not depend on $a$. Hence an application of Proposition 4.1 would yield the same bound on each Gaussian width appearing inside the summation in the statement of Lemma 8.11. From this we can deduce Proposition 8.9 in a straightforward manner. The detailed computation is given below. In the remainder of the proof we will use $C$ to denote any positive constant depending at most on $c$ whose exact value may change from one line to the next.

Applying Proposition 4.1 with $k_{0}=\lfloor-\log_{2}2t\rfloor$ and $k_{1}=-\lceil\log_{2}\nu\rceil$ where $\nu=t/m\vee m^{-10}$ and using Lemma 8.12 subsequently to bound the relevant terms (see Remark 8.5), we get

[TABLE]

Now recalling the definition of $\mathcal{L}_{m}(\cdot)$, we can write

[TABLE]

where in the last inequality we used the fact that $C_{m,n,\delta,t}\leq C(1+\delta)\log(\mathrm{e}m)$ since $t\leq 1$ and $m/n$ is assumed to be bounded by a constant. The last two displays therefore imply

[TABLE]

where in the final step we used the fact that $\delta\in(0,n]$ as well as $\max\{m/n,n/m\}\leq c$. The proposition now follows from summing this bound over $k$ as in Lemma 8.11. ∎

The thing that remains to be done is the proof of Lemma 8.12. An important ingredient is the following weaker analogue for the general case.

Lemma 8.13**.**

Let $k,m,n$ be positive integers with $1\leq k\leq m$ (not to be confused with the parameters in Lemmata 8.11 – 8.12). Also let $t\leq 1$ and $u,v,\tau>0$. Then there exists a $\tau$ subspace cover $S_{\tau}$ of $\mathcal{A}(m,n,u,v,t)$, depending on $m,n,k,u,v$ and $t$ in addition to $\tau$, and a universal constant $C>0$ such that

[TABLE]

when $k<m$, whereas for $k=m$

[TABLE]

Here

[TABLE]

Remark 8.7**.**

Lemma 8.13, by itself, is not sufficient to prove Lemma 8.12. To see this, let us plug in $n=m/a^{2}$ and $u=2ta$ in the expression for $J_{k}$. One can easily check that while this makes $J_{k}$ free from $a$, the principal terms in the bounds on the dimension and cardinality do not attain the required $\sqrt{m}$-scaling for any choice of $k$.

In the course of proving Lemma 8.13, we will repeatedly use a subdivision scheme based on the value of either ${\mathrm{TV}}_{\mathrm{row}}$ or ${\mathrm{TV}}_{\mathrm{col}}$. We will also use it in the proof of Lemma 8.12 and therefore describe it here in a general setting. Let us point out that a very similar scheme was described in Section 5.1 in the context of proving Theorem 2.1.

**A greedy partitioning scheme:  **Consider a set $\mathcal{S}$ and a function $T:\cup_{n\in\mathbb{N}}\mathcal{S}^{n}\mapsto\mathbb{R}_{\geq 0}$ satisfying $T(AB)\geq T(A)+T(B)$ for all $A,B\in\cup_{n\in\mathbb{N}}\mathcal{S}^{n}$ where $AB$ denotes the concatenation of $A$ and $B.$ Also suppose for any singleton $s\in S,$ the function $T$ satisfies $T(s)=0$. To relate this to a concrete example, the reader may consider the case where $\mathcal{S}=\mathbb{R}^{m}$ so that $\mathcal{S}^{n}\equiv\mathbb{R}^{m\times n}$ and $T$ is the function ${\mathrm{TV}}_{\mathrm{row}}$. Now for any $\epsilon>0$, the $(T,\epsilon)$ scheme subdivides an element $U$ of $\cup_{n\in\mathbb{N}}\mathcal{S}^{n}$ as $U_{1}U_{2}\cdots U_{K}$ such that $T(U_{i})\leq\epsilon$ for all $i\in[K]$. This is achieved in several steps of binary division as follows. In the first step, we check whether $T(U)\leq\epsilon.$ If so, then stop and output $U.$ Else, divide $U$ as $U^{\prime}_{1}U^{\prime}_{2}$ into two almost equal parts. This means $|U^{\prime}_{1}|=\lfloor|U|/2\rfloor$ and $|U^{\prime}_{2}|=|U^{\prime}|-|U^{\prime}_{1}|.$ In each step, we have a representation of $U$ of the form $U^{\prime}_{1}U^{\prime}_{2}\cdots U^{\prime}_{K^{\prime}}.$ We consider each $i\in[K^{\prime}]$ such that $T(U^{\prime}_{i})>\epsilon$ and subdivide $U^{\prime}_{i}$ into two almost equal parts. We repeat this procedure until each part $U^{\prime}$ in the current representation satisfies $T(U^{\prime})\leq\epsilon$.

Suppose that $|U|=n$. The subdivision of $U$ produced by the $(T,\epsilon)$ scheme corresponds to a partition of $[n]$ into contiguous blocks, say, $P_{U;T,\epsilon}$. Let $|P_{U;T,\epsilon}|$ denote the number of blocks of the partition $P_{U;T,\epsilon}.$ Now for $t>0$, let $\mathcal{P}(t,n,\epsilon,T)$ denote the set of partitions $\{P_{U;T,\epsilon}:U\in\mathcal{S}^{n},T(C)\leq t\}$. A key ingredient in the proof of Lemma 8.13 (and subsequently Lemma 8.12) is the following universal upper bound on the cardinality of $P_{U;T,\epsilon}$.

Lemma 8.14**.**

Then for the $(T,\epsilon)$ division scheme we have

[TABLE]

The proof of Lemma 8.14 is very similar to that of Lemma 5.4. Nevertheless, for the sake of completeness, we provide its proof in the appendix (see Section 10.2). We also defer the proof of Lemma 8.13 to the end of this subsection and finish the proof of Lemma 8.12 assuming it.

Proof of Lemma 8.12.

Take any $\theta\in\mathcal{A}_{a}$ and fix $\epsilon\in(0,1)$ whose precise value based on $\tau$ would be chosen later. Let us denote the $m^{\prime}\times n^{\prime}$ two dimensional grid (graph) by $L_{m^{\prime},n^{\prime}}$ and subdivide $L_{m,m/a^{2}}$ as

[TABLE]

where ${\mathrm{TV}}_{\mathrm{col}}(\theta_{|R_{i}})\leq\epsilon$ for all $i\in[K]$ and $K\leq\log_{2}(4m)(1+{\mathrm{TV}}_{\mathrm{col}}(\theta)\epsilon^{-1})$. We achieve this by applying the $({\mathrm{TV}}_{\mathrm{col}},\epsilon)$ division scheme to the rows of $\theta$ (see Lemma 8.14). Denoting the set of all possible partitions of $L_{m,m/a^{2}}$ obtained in this manner by $\mathcal{P}$, we deduce

[TABLE]

Corresponding to the partition $P=P(\theta)$ in (8.13), let $S_{P,\rm row}$ denote the linear subspace of $\mathbb{R}^{m\times m/a^{2}}$ comprising only matrices having identical rows in each $R_{i}$. It is clear that the orthogonal projection of $\theta$ onto $S_{P,\rm row}$ is given by

[TABLE]

where each row of $\widetilde{\theta}_{i}\coloneqq\widetilde{\theta}_{|R_{i}}$ is equal to the average row of $\theta_{R_{i}}$. By repeated application of Lemma 10.3 (stated and proved in the appendix), we obtain

[TABLE]

Also by standard properties of orthogonal projections, it follows that $\|\widetilde{\theta}\|_{2}\leq\|\theta\|_{2}\leq t$. We further claim that $\widetilde{\theta}\in\mathcal{A}_{a}\equiv\mathcal{A}(m,\frac{m}{a^{2}},2ta+\>\delta,\>t\sqrt{\frac{m}{n}}+\delta,t)$. Hence to establish this claim we only need to show that ${\mathrm{TV}}_{\mathrm{row}}(\widetilde{\theta})\leq{\mathrm{TV}}_{\mathrm{row}}(\theta)$ and ${\mathrm{TV}}_{\mathrm{col}}(\widetilde{\theta})\leq{\mathrm{TV}}_{\mathrm{col}}(\theta)$. We can obtain the first inequality as follows:

[TABLE]

For the second inequality we just apply Lemma 10.4 (stated and proved in the appendix section) to each column of $\theta$.

In the rest of the article we call a subset of $L_{m^{\prime}\times n^{\prime}}$ a subgrid if it is a product of subsets (as opposed to only subintervals) of $[1,m^{\prime}]\cap\mathbb{N}$ and $[1,n^{\prime}]\cap\mathbb{N}$ respectively. We will now regroup $R_{i}$’s into several subgrids. For any positive integer $\ell$ such that $2^{\ell}\leq{2}m$, define the set S_{\ell}\coloneqq\big{\{}i\in[K]:2^{\ell-1}\leq\mathrm{n}_{\mathrm{row}}(R_{i})<2^{\ell}\big{\}} and let $B_{\ell}$ be the vector which is the sorted version of $S_{\ell}$. Now consider the subgrid of $L_{m,m/a^{2}}$

[TABLE]

where $K_{\ell}\coloneqq|B_{\ell}|$. In words, $R^{\ell}$ comprises the rectangles $R_{i}$, in order, whose number of rows lies between $2^{\ell-1}$ and $2^{\ell}$. It is clear that $R^{1},R^{2},\dots,R^{L}$ are disjoint subgrids of $L_{m,n}$ where $L\leq\log_{2}(2m)$. Let us also denote $\widetilde{\theta}_{|R^{\ell}}$ by $\widetilde{\theta}^{\ell}$. Notice that if the matrices $\widehat{\theta}^{1},\widehat{\theta}^{2},\dots,\widehat{\theta}^{L}$ satisfy $\|\widetilde{\theta}^{\ell}-\widehat{\theta}^{\ell}\|\leq\sqrt{m}\epsilon$ for all $\ell\in[L]$ and $\widehat{\theta}\in\mathbb{R}^{m\times m/a^{2}}$ is such that $\widehat{\theta}_{|R^{\ell}}=\widehat{\theta}^{\ell}$ for all $\ell\in[L]$, then we have

[TABLE]

We now choose $\epsilon$ by requiring this approximation error to be $\tau$, i.e., by setting $\epsilon=\tau/\sqrt{2\>m\log_{2}(4m)}$ (notice that $1/4m^{2}\leq\epsilon\leq 1/\sqrt{m}$ when $\tau\in[1/m,1]$). Therefore if $\mathcal{S}_{\tau,P}^{\ell}$ is a $\sqrt{m}\epsilon$ subspace cover for the family $\mathcal{A}_{\ell,P}^{*}$ (say) of matrices $\widetilde{\theta}^{\ell}$ corresponding to $P\in\mathcal{P}$ and $\ell\in[L]$, we can immediately obtain a $\tau$ subspace cover $S_{\tau}$ for $\mathcal{A}_{a}$ satisfying:

[TABLE]

and

[TABLE]

Now fix a $P\in\mathcal{P}$ and let $\Theta^{\ell}$ denote the matrix formed by the first (or any) rows of $\widetilde{\theta}_{B_{\ell}(1)},\widetilde{\theta}_{B_{\ell}(2)},\dots,$ $\widetilde{\theta}_{B_{\ell}(k_{\ell})}$ in order, i.e., the rows of $\widetilde{\theta}^{\ell}$ that are potentially distinct. We claim that

[TABLE]

The constraints on the number of rows and columns of $\Theta^{\ell}$ as well as $\|\Theta^{\ell}\|$ are clear. For the remaining constraints first observe that $\widetilde{\theta}^{\ell}\in\mathcal{A}(\mathrm{n}_{\mathrm{row}}(R^{\ell}),\frac{m}{a^{2}},2ta+\>\delta,\>t\sqrt{\frac{m}{n}}+\delta,t)$ (the only non-obvious part is the bound on ${\mathrm{TV}}_{\mathrm{col}}(\widetilde{\theta}^{\ell})$ which follows from the triangle inequality). From the definition of $\Theta^{\ell}$ it is immediate that

[TABLE]

Therefore the bounds on ${\mathrm{TV}}_{\mathrm{col}}(\Theta^{\ell})$ and ${\mathrm{TV}}_{\mathrm{row}}(\Theta^{\ell})$ follow from the similar bounds for $\widetilde{\theta}^{\ell}$ and the fact that $\mathrm{n}_{\mathrm{row}}(R_{B_{\ell}(i)})\geq 2^{\ell-1}$ for each $i\in[K_{\ell}]$.

Further notice that since $\mathrm{n}_{\mathrm{row}}(R_{B_{\ell}(i)})<2^{\ell}$ for each $i\in[K_{\ell}]$, we have $\|\widetilde{\theta}^{\ell}-\widehat{\theta}^{\ell}\|\leq 2^{\ell/2}\|\Theta^{\ell}-\widehat{\Theta}^{\ell}\|$ where $\widehat{\theta}^{\ell}$ comprises repetitions of the rows of $\widehat{\Theta}^{\ell}$ in the same way as $\widetilde{\theta}^{\ell}$ comprises repetitions of the rows of $\widetilde{\Theta}^{\ell}.$ Therefore any $2^{-\ell/2}\sqrt{m}\epsilon$ subspace cover $\mathcal{S}_{\epsilon}^{\ell}$ for $\mathcal{A}_{a,\ell}$ induces a $\sqrt{m}\epsilon$ subspace cover $\mathcal{S}_{\tau,P}^{\ell}$ for $\mathcal{A}^{*}_{\ell,P}$. Our next claim is about a uniform upper bound on $\max_{S\in\mathcal{S}_{\epsilon}^{\ell}}\dim(S)$ and $|\mathcal{S}_{\epsilon}^{\ell}|$ for some particular choice of $\mathcal{S}_{\epsilon}^{\ell}$ and hence that of $\max_{S\in\mathcal{S}_{\tau,P}^{\ell}}\dim(S)$ and $|\mathcal{S}_{\tau,P}^{\ell}|$ as well.

Claim 8.2**.**

There is a choice of $\mathcal{S}_{\epsilon}^{\ell}$ for any $\ell\in\mathbb{N}_{>0}$ and $\epsilon\in[1/m^{2},1/\sqrt{m}]$ such that for some universal constant $C>0$,

[TABLE]

where we recall from the statement of Lemma 8.12 that $\mathcal{L}(x)=x\log(\mathrm{e}\log(\mathrm{e}m)^{2}x)$ and $x^{2\downarrow}=x+x^{2}$.

Claim 8.2 follows directly from Lemma 8.13 when we choose $k$ in an appropriate manner. The complete proof is given after the current proof.

Concluding the proof. In the remainder of the proof we will use $C$ to denote any positive, universal constant whose exact value may change from one line to the next. Using Claim 8.2 let us first bound

[TABLE]

where we used the fact that $\tfrac{1}{\sqrt{m}\epsilon^{2}}=\tfrac{\sqrt{m}\log_{2}(4m)}{\tau^{2}}$ (recall the choice of $\epsilon$ after (8.17) and also the definition of $C_{m,n,\delta,t}$ from the statement of Lemma 8.12). On the other hand, since $\epsilon\leq 1/\sqrt{m}$, we can bound $\log|\mathcal{P}|$ in view of (8.14) as

[TABLE]

Since $\mathcal{L}(x)\geq x$ for all $x\geq 1$, we deduce by combining the previous two displays and subsequently plugging them into (8.18)–(8.19):

[TABLE]

Proof of Claim 8.2.

The “main” contribution in the bounds on $\log|\mathcal{S}_{\epsilon}^{\ell}|$ and $\max_{S\in\mathcal{S}_{\epsilon}^{\ell}}\dim(S)$ given by Lemma  8.13 comes from

[TABLE]

where

[TABLE]

(recall the statement of Lemma 8.13 and (8.20)). Therefore, as already mentioned in the proof of Lemma 8.12, we will apply Lemma 8.13 for some $k\in[K_{\ell}]$ so that $J_{k,\ell}^{*}$ has a small value. In the rest of the proof we will use $C$ to denote an unspecified but universal positive constant whose value may change from one instant to the next. Using the simple fact $\sqrt{x+y}\leq\sqrt{x}+\sqrt{y}$, we can bound $J_{k,\ell}^{*}$ as follows:

[TABLE]

where

[TABLE]

Now let us consider two cases separately based on whether $\sqrt{K_{\ell}}2^{-\ell/2}\sqrt{m}\epsilon$ is smaller or larger than 1. Recall that $2^{-\ell/2}\sqrt{m}\epsilon$ is the covering radius in question, and the condition above is equivalent to $K_{\ell}$ being smaller or larger than the inverse of the covering radius squared.

Case 1: $K_{\ell}\leq\frac{2^{\ell}}{m\epsilon^{2}}$. In this case we choose $k=K_{\ell}$ so that Lemma 8.13 and (8.21) together give us

[TABLE]

where

[TABLE]

Now using

[TABLE]

for the first term inside the parenthesis in (8.25) (recall the definition of $K_{\ell}$ and $K$ from the proof of Lemma 8.12) and using $K_{\ell}\leq\frac{2^{\ell}}{m\epsilon^{2}}$ for the second, we get

[TABLE]

Further noticing that $\epsilon\leq 1/\sqrt{m}$, so that $\tfrac{1}{\epsilon}\leq\tfrac{1}{\sqrt{m}\epsilon^{2}}$, we obtain

[TABLE]

(recall that $x^{2\downarrow}=x+x^{2}$). On the other hand we have $K_{\ell}=k\leq J_{k,\ell}$ for $C>1$. Plugging these bounds into the right hand side of (8.24) and rewriting the expression in terms of $\mathcal{L}_{m}(x)=x\log(\mathrm{e}\log(\mathrm{e}m)^{2}x)$ we obtain

[TABLE]

where we used the fact that $\log(C\mathrm{e}\log(\mathrm{e}m)^{2}x)\leq C\log(\mathrm{e}\log(\mathrm{e}m)^{2}x)$ for all $x\geq 1$ and large enough $C$.

*Case 2: $K_{\ell}\geq\frac{2^{\ell}}{m\epsilon^{2}}$. * Notice that in this case we can choose $k=\lfloor\frac{2^{\ell}}{m\epsilon^{2}}\rfloor$ and Lemma 8.13 gives us

[TABLE]

We will show below that the right hand side of (8.27) also serves as an upper bound for $J_{k,\ell}^{*}$ and $K_{\ell}$, and consequently the upper bound in (8.28) holds in this case as well, thus proving the claim. To this end we will use the bounds (8.22) and (8.23). First observe that the bound on $J_{k,\ell}$ is same as in the previous case since the only bounds we used there were $k\leq K_{\ell}$ and $k\leq\frac{2^{\ell}}{m\epsilon^{2}}$, both of which are valid in this case. On the other hand, $C_{\epsilon,k,\ell}$ can be bounded by

[TABLE]

Since $K_{\ell}\geq\frac{2^{\ell}}{m\epsilon^{2}}$ and $\epsilon\leq 1/\sqrt{m}$, we have

[TABLE]

(cf. the right hand side of (8.27)). Similarly we can bound

[TABLE]

Plugging these bounds into the (8.30) we get

[TABLE]

where used the simple fact that $x^{3/2}\leq x^{2\downarrow}$. Combined with (8.27) and the discussion preceding the display (8.30), this yields us a similar upper bound for $J_{k,\ell}^{*}$. ∎

We are only left with the proof of Lemma 8.13.

Proof of Lemma 8.13.

The proof is split into two parts. In the first part we try to construct, for any given $\theta\in\mathcal{A}(m,n,u,v,t)$, another matrix $\widehat{\theta}$ satisfying $\|\theta-\widehat{\theta}\|\leq\tau$ such that $\widehat{\theta}$ is piecewise constant on rectangles with as few blocks as possible. These blocks define a partition $P$ of $L_{m,n}$ and let $\mathcal{P}$ denote the set of all such partitions. It is then clear that $S_{\tau}\coloneqq\{S_{P}:P\in\mathcal{P}\}$ forms a $\tau$ subspace cover of $\mathcal{A}(m,n,u,v,t)$ (see the proof of Theorem 2.1 in Section 5 for the notation and similar notions). In the second and the final part we bound $\max_{P\in\mathcal{P}}|P|$ and $|\mathcal{P}|$ which, in view of the definition above, yield the desired upper bounds on $\max_{S\in\mathcal{S}_{\tau}}\dim(S)$ and $|\mathcal{S}_{\tau}|$.

**Approximating $\theta$ by a piecewise constant matrix. ** This part consists of three steps. In the “zeroth” step, we divide $\theta$ equally into $k$ submatrices by horizontal divisions. We do not choose, a priori, any specific value of $k$ which is the reason why our final bound depends on $k$. Then in step 1, each of these submatrices is divided into submatrices by vertical divisions which are again subdivided in step 2 by horizontal divisions. The rectangles corresponding to these submatrices will be the final level sets of $\widehat{\theta}$. We now elaborate the steps.

*Step 0: Horizontal Divisions. * Fix a positive integer $1\leq k\leq m$ and divide $L_{m,n}$ into $k$ submatrices as follows:

[TABLE]

where each $R_{i}$ has either $\lceil m/k\rceil$ or $\lfloor m/k\rfloor$ many rows. We want to stress that we use the same partitioning for every $\theta$ in this step.

*Step 1: Vertical Divisions. * Next we want to subdivide each $R_{i}$ (where $i\in[k]$) by making $j_{i}$ many vertical divisions:

[TABLE]

such that ${\mathrm{TV}}_{\mathrm{row}}(\theta_{|R_{i,j}})\leq\tau_{k}$ for all $j\in[j_{i}]$ and some $\tau_{k}>0$ to be chosen shortly. We can do this by the $({\mathrm{TV}}_{\mathrm{row}},\tau_{k})$ scheme applied to the columns of $\theta_{i}$ so that Lemma 8.14 gives us the bounds

[TABLE]

Replacing each element in every row of $\theta_{i,j}\coloneqq\theta_{|R_{i,j}}$ with the corresponding row mean, we then obtain a new matrix

[TABLE]

By construction, each $\widetilde{\theta}_{i,j}$ has identical columns. Finally, let us define

[TABLE]

From the Cauchy-Schwarz inequality, it is clear that $\|\widetilde{\theta}\|\leq\|\theta\|$. One important observation we need make at this point is that while this averaging procedure might increase the value of ${\mathrm{TV}}_{\mathrm{col}}(\widetilde{\theta})$, it does not increase the value of ${\mathrm{TV}}_{\mathrm{col}}(\widetilde{\theta}_{i,j})$ for any $i$ and $j$. Indeed by a computation exactly similar to that performed in (8.6) we get

[TABLE]

Let us now try to bound $\|\theta-\widetilde{\theta}\|$. To this end notice that

[TABLE]

where in the final step we used Lemma 10.3. Since ${\mathrm{TV}}_{\mathrm{row}}(\theta_{i,j})\leq\tau_{k}$, we can then deduce

[TABLE]

Setting $\tau_{k}=\tau/2\sqrt{nk}$, we get $\|\theta-\widetilde{\theta}\|_{2}\leq\tau/2$.

*Step 2: Horizontal Divisions. *In this step, we are going to make horizontal divisions within each $R_{i,j}$ obtained from step 1 so that the total variation of columns of $\widetilde{\theta}_{i,j}$ restricted to each subdivision is smaller than some fixed, small number. To this end fix $\tau_{k}^{\prime}>0$ whose exact value will be chosen later. Now use the $({\mathrm{TV}}_{\mathrm{col}},\tau_{k}^{\prime})$ scheme applied to the rows of $R_{i,j}$ to obtain the following subdivision:

[TABLE]

where, with $\widetilde{\theta}_{i,\ell;j}\coloneqq\widetilde{\theta}_{|R_{i,\ell;j}}$, ${\mathrm{TV}}_{\mathrm{col}}[\widetilde{\theta}_{i,\ell;j}]\leq\tau_{k}^{\prime}$ for all $\ell\in[\ell_{i,j}]$. From Lemma 8.14 we can deduce

[TABLE]

Like in the definition of $\widetilde{\theta}_{i,j}$, we now replace every element in each column of $\widetilde{\theta}_{i,\ell;j}$ (recall at this point that $\widetilde{\theta}_{i,j}$ and hence $\widetilde{\theta}_{i,\ell;j}$ has identical columns) with the corresponding column mean and obtain a new matrix

[TABLE]

By construction, $\widehat{\theta}_{i,\ell;j}$ is a constant matrix. Let $\widehat{\theta}\in\mathbb{R}^{m\times n}$ be such that $\widehat{\theta}_{|R_{i,\ell;j}}=\widehat{\theta}_{i,\ell;j}$. By the Cauchy-Schwarz inequality we have $\|\widehat{\theta}\|\leq\|\widetilde{\theta}\|\leq\|\theta\|$.

We now want to bound the distance between $\widetilde{\theta}$ and $\widehat{\theta}$. Notice that, since the columns of $\widetilde{\theta}_{i,\ell;j}$ are identical, we get from Lemma 10.3

[TABLE]

for every $j^{\prime}\in\mathrm{n}_{\mathrm{col}}(\widetilde{\theta}_{i,j})=\mathrm{n}_{\mathrm{col}}(\widetilde{\theta}_{i,\ell;j})$. Summing over $i,j,\ell$ and $j^{\prime}$, we then deduce

[TABLE]

Let us choose

[TABLE]

so that $\|\widetilde{\theta}-\widehat{\theta}\|_{2}\leq\tau/2$ and hence

[TABLE]

**Counting the number of possible partitions for any $\theta$. ** Fix any vertical division of $\theta$ obtained in step 1. Now summing (8.35) over all $i$ and $j$ we get

[TABLE]

where we used the following fact

[TABLE]

On the other hand (8.36) allows us to deduce a naive lower bound on $\tau_{k}^{\prime}$ as follows:

[TABLE]

Plugging this into (8.37) we get for a universal constant $C>0$,

[TABLE]

where $n_{piece}(\widehat{\theta})$ is the total number of rectangular level sets of $\widehat{\theta}$ and $J\coloneqq\sum_{i\in[k]}j_{i}$. From now onwards we will implicitly assume that $C$ is a positive, universal constant whose exact value may vary from one line to the next.

Therefore the number of tuples $(\ell_{1,1},\ell_{1,2},\ldots,\ell_{k,j_{k}})$ satisfying (8.38) is at most

[TABLE]

Similarly, in order to bound $J$ we sum (8.31) over all $i$ to obtain

[TABLE]

where in the final step we used $\tau_{k}=\tau/2\sqrt{nk}$ (see the end of step 1 in the previous part).

It remains to count the number of possible vertical divisions in step 1. To this end let us fix a tuple $(j_{1},j_{2},\ldots,j_{k})$ satisfying $\sum_{i\in[k]}j_{i}\leq J_{k}$. The number of possible vertical divisions in this case is bounded by $\prod_{i\in[k]}n^{j_{i}}=n^{J_{k}}$. On the other hand, in view of (8.31) and (8.6), the number of tuples $(j_{1},j_{2},\ldots,j_{k})$ is bounded by the number of nonnegative integral solutions to the inequality $\sum_{i\in[k]}j_{i}\leq J_{k}$ which in turn is bounded by $(J_{k})^{k}$. Putting all of these together with (8.39) and (8.6), we can now deduce the following upper bound on the total number of possible partitions for any $\theta\in\mathcal{A}(m,n,u,v,t)$:

[TABLE]

From this and (8.38) we can derive the bound for any $1\leq k<m$. For the second bound, that is when $k=m$, recall that the second summand in the right hand side of (8.37) comes from the horizontal division conducted in step 2. Since this step becomes void for $k=m$, the required bound follows in exactly similar fashion with $J_{k}$ replacing $J_{k}+\sqrt{J_{k}}\tfrac{v\sqrt{m}}{\tau\sqrt{k}}$. ∎

9 Proof of Theorem 2.7

To prove Theorem 2.7 we apply the general machinery developed in chatterjee2015high with suitable modifications. Let us define $w=y-\overline{y}$ to be the centered data matrix, $w^{*}=\theta^{*}-\overline{\theta^{*}}$ to be the centered ground truth matrix and let

[TABLE]

Also, for any $V\geq 0,$ let $\widehat{w}_{V}$ denote the Euclidean projection of $w$ onto the convex set $K_{n}^{0}(V).$ Recall that $K_{n}^{0}(V)\coloneqq\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}(\theta)\leq V,\,\overline{\theta}=0\}\,.$

9.1 Sketch of Proof

To show that $\widehat{\theta}_{\mathrm{notuning}}$ is a good estimator of $\theta^{*}$ it clearly suffices to show that $\widehat{w}$ is a good estimator of $w^{*}.$ If we knew $\mathrm{TV}(\theta^{*})=\mathrm{TV}(w^{*})=V^{*}$, a similar argument as in the proof of Theorem 2.1 would tell us that $\widehat{w}_{V^{*}}$ attains the $\widetilde{O}(\frac{V^{*}}{\sqrt{N}})$ rate that we desire. Of course, the aim here is to get the same rate without knowing $V^{*}$ and $\sigma.$ One part of our proof deals with showing that using $\widehat{\sigma}$ in the definition of our estimator is not much worse than if we knew $\sigma$ and used it in defining our estimator. This is shown by showing that $\widehat{\sigma}\approx\sigma$ using a concentration of measure argument where “$\approx$” is a somewhat informal notation conveying the meaning of approximately equal to.

To analyze the risk of $\widehat{w},$ a natural first step is to decompose the risk as follows:

[TABLE]

Here we used the elementary inequality $\|a+b\|^{2}\leq 2\|a\|^{2}+2\|b\|^{2}$. The above decomposition has a natural interpretation as twice the sum of the ideal risk (achievable when $V^{*}$ is known) and an excess risk due to not knowing $V^{*}$ and $\sigma.$ The main task therefore is to upper bound the excess risk term $\|\widehat{w}-\widehat{w}_{V^{*}}\|^{2}.$

We now need to look at two different cases. The first case is when $\widehat{w}\neq\textbf{0}.$ In this case we first show that the minimum of the optimization problem defined in (9.1) is attained on the boundary. This would mean we have $\|\widehat{w}-w\|^{2}=(n^{2}-1)\widehat{\sigma}^{2}\approx(n^{2}-1)\sigma^{2}.$ Letting $\widehat{V}=\mathrm{TV}(\widehat{w})$, a simple geometric argument also shows that $\widehat{w}_{\widehat{V}}=\widehat{w}.$ Thus, both $\widehat{w}_{V^{*}}$ and $\widehat{w}$ are Euclidean projections onto $K_{n}^{0}(V)$ for two possibly different choices of $V.$ Thus, we can now use standard characterizations of Euclidean projections onto convex sets (content of Lemma 9.1) for both $\widehat{w}_{V^{*}}$ and $\widehat{w}$ to obtain a bound on the excess risk as follows:

[TABLE]

Since $\|\widehat{w}-w\|^{2}\approx(n^{2}-1)\sigma^{2}$ we can then conclude

[TABLE]

Further, since $\widehat{w}_{V^{*}}$ is known to be a good estimator of $w^{*}$ we can write

[TABLE]

where the last approximation is again by a simple concentration of measure argument. The last three displays then suggest that $\widehat{w}$ is close to $\widehat{w}_{V^{*}}.$ Quantifying the last three displays gives us the desired upper bound on the excess risk.

The second case is when $\widehat{w}=\textbf{0}.$ By definition we have $\|\widehat{w}\|^{2}\leq(n^{2}-1)\widehat{\sigma}^{2}\approx(n^{2}-1)\sigma^{2}.$ Since $\textbf{0}\in K_{n}^{0}(V^{*})$ and $\widehat{w}_{V^{*}}$ is the projection of $w$ onto $K_{n}^{0}(V^{*})$, a standard fact about Euclidean projections onto convex sets gives $\langle w-\widehat{w}_{V^{*}},\widehat{w}_{V^{*}}\rangle\geq 0.$ This implies

[TABLE]

The rest of the proof then follows similarly as in the previous case.

9.2 Full Proof

While proving Theorem 2.7 we will prove a few intermediate results. Our first lemma is a basic fact about Euclidean projections onto $K_{n}^{0}(V)$ for two different choices of $V.$ This also appears as Lemma $5.1$ in chatterjee2015high. For the sake of completeness, we give a proof in Subsection LABEL:sec:projfactproof of the appendix.

Lemma 9.1**.**

Let $y\in\mathbb{R}^{n\times n}$ and recall $K_{n}^{0}(V)\coloneqq\{\theta\in\mathbb{R}^{n\times n}:\mathrm{TV}(\theta)\leq V,\overline{\theta}=0\}.$ Let $V_{1}>V_{2}\geq 0$ and let $\pi_{1}(y),\pi_{2}(y)$ be the Euclidean projection of $y$ onto the convex sets $K_{n}^{0}(V_{1}),K_{n}^{0}(V_{2})$ respectively. Then we have the following inequality:

[TABLE]

Our next lemma is the following pointwise inequality.

Lemma 9.2**.**

Let $w=y-\overline{y}\textbf{1}$ be the centered version of $y.$ For any $V\geq 0,$ let $\widehat{w}_{V}$ denote the projection of $w$ onto the convex set $K_{n}^{0}(V).$ Let

[TABLE]

Then we have the following pointwise inequality;

[TABLE]

Proof.

Let us first consider the case when $\widehat{w}\neq\textbf{0}.$ Define $\widehat{V}\coloneqq\mathrm{TV}(\widehat{w}).$ We claim that $\widehat{w}_{\widehat{V}}=\widehat{w}$ and further

[TABLE]

To prove the above claim, suppose $\widehat{w}_{\widehat{V}}\neq\widehat{w}.$ Then we have $\|w-\widehat{w}_{\widehat{V}}\|^{2}<\|w-\widehat{w}\|^{2}\leq(n^{2}-1)\widehat{\sigma}^{2}$ because of uniqueness of Euclidean projections onto convex sets. Therefore, we have $\|w-\widehat{w}_{\widehat{V}}\|^{2}<(n^{2}-1)\widehat{\sigma}^{2}$ and $\|w-\textbf{0}\|^{2}>(n^{2}-1)\widehat{\sigma}^{2}$ by assumption. Let us now draw a line segment connecting $\widehat{w}_{\widehat{V}}$ to the origin and select the point which cuts the boundary of the $\sqrt{(n^{2}-1)}\,\widehat{\sigma}$ ball around $w$ and call it $w^{{\rm bdry}}.$ Then by construction we have

[TABLE]

Since $w$ has zero mean, it is not hard to see that $\widehat{w}_{\widehat{V}}$ has mean zero as well because $\widehat{w}_{\widehat{V}}$ is the Euclidean projection of $w$ onto $K_{n}^{0}(\widehat{V}).$ Therefore any point falling on the line segment between $\widehat{w}_{\widehat{V}}$ and the origin also must have mean zero, including $w^{{\rm bdry}}.$ Thus $w^{{\rm bdry}}$ is feasible for the optimization problem defined in (9.2). Together with (9.4) this contradicts the definition of $\widehat{w}.$ Therefore $\widehat{w}_{\widehat{V}}$ must be equal to $\widehat{w}$ and (9.3) must hold.

Letting $V^{*}=\mathrm{TV}(\theta^{*})$, we can now write

[TABLE]

where we have applied Lemma 9.1 in the first inequality and used (9.3) in the last equality.

Now let us consider the case when $\widehat{w}=\textbf{0}.$ In this case we can write

[TABLE]

The first inequality uses Lemma 9.1 and the second equality follows from the definition of $\widehat{w}_{V^{*}}$ upon observing that $\bm{0}\in K_{n}^{0}(V^{*})$. Finally the third inequality uses the fact that $\|w\|^{2}\leq(n^{2}-1)\widehat{\sigma}^{2}$ since $\widehat{w}=\textbf{0}$. This finishes the proof of the lemma. ∎

Our next result is a proposition which gives a pointwise upper bound to the squared loss.

Proposition 9.3**.**

Let $V^{*}=\mathrm{TV}(\theta^{*}).$ Let $w=y-\overline{y}\textbf{1}$ and $w^{*}=\theta^{*}-\overline{\theta^{*}}\textbf{1}$ be the centered versions of $y$ and $\theta^{*}$ respectively. Also let $\widehat{w}_{V^{*}}$ denote the Euclidean projection of $w$ onto $K_{n}^{0}(V^{*}).$ Then the following pointwise risk inequality holds:

[TABLE]

Proof.

By definition of $\widehat{\theta}$ and Pythagorean theorem we have

[TABLE]

We can now use Lemma 9.2 and the triangle inequality to write

[TABLE]

Let us now bound the third term above on the right side.

[TABLE]

We now observe that for any mean zero matrix $v$, we can write

[TABLE]

The last two displays then imply that

[TABLE]

Further, from the basic inequality $\|w-\widehat{w}_{V^{*}}\|^{2}\leq\|w-w^{*}\|^{2}$ we can conclude

[TABLE]

The last display along with  (9.5), (9.2) and  (9.7) finish the proof of the proposition. ∎

We are now in a position to finally prove Theorem 2.7.

Proof of Theorem 2.7.

It suffices to take expectation over the four terms which consists in the upper bound given in Proposition 9.3. We now sequentially bound the expectation of these terms. We will use $C$ to denote a positive, universal constant whose exact value may change from one line to the next.

The first term is just $8\sigma$ times the Gaussian width of $K_{n}^{0}(2V^{*})$ and we can use 5.2 to upper bound it. As for the second term, it is clear that

[TABLE]

Also we observe that $\frac{\|w-w^{*}\|^{2}}{\sigma^{2}}=\sum_{i=1}^{n}\sum_{j=1}^{n}(Z_{ij}-\overline{Z})^{2}\approx\chi^{2}_{n^{2}-1}.$ This is a standard fact about standard normal random variables. Therefore we can write

[TABLE]

where the first inequality follows from the Cauchy Schwartz inequality and the last equality follows because $\text{Var}(\chi^{2}_{k}))=2\>k$ for any positive integer $k.$

Next we bound $\mathbb{E}|\widehat{\sigma}^{2}-\sigma^{2}|.$ We can write

[TABLE]

Recalling the definition of $\widehat{\sigma}$ we have

[TABLE]

Thus we can write

[TABLE]

Now, since $\mathrm{TV}(Z)$ is a sum of $N(0,2)$ random variables it is easy to check that $\mathbb{E}\mathrm{TV}(Z)=\frac{4\>n\>(n-1)}{\sqrt{\pi}}.$ Also by Lemma 9.4 we can upper bound the variance of $\mathrm{TV}(Z)$ to get

[TABLE]

Taking expectation on both sides of (9.9) we obtain

[TABLE]

Using (9.8), the last display and the Cauchy-Schwarz inequality to bound $\mathbb{E}|\widehat{\sigma}-\sigma|$, we can deduce

[TABLE]

Collecting the bounds we have obtained in this proof for the four terms comprising the upper bound given in Proposition 9.3, we can conclude that

[TABLE]

This finishes the proof of Theorem 2.7. ∎

It only remains to prove the following lemma.

Lemma 9.4**.**

There exists a universal constant $C>0$ such that

[TABLE]

Proof.

Expanding $\mathrm{Var}(\mathrm{TV}(\mathbf{Z}))$ we get

[TABLE]

where in the second step we used the observation that $\mathrm{Cov}(|\Delta_{e}\mathbf{Z}|,|\Delta_{e^{\prime}}\mathbf{Z}|)=0$ for all non-adjacent $e,e^{\prime}$, i.e., $e,e^{\prime}$ which do not share any vertex. Here $e^{\prime}\sim e$ means the edges $e,e^{\prime}$ are adjacent. Since each edge $e$ is adjacent to finitely many edges (including $e$ itself) we get from (9.10) that $\mathrm{Var}(\mathrm{TV}(\mathbf{Z}))\leq C|E_{n}|$ for some universal constant $C>0$. The lemma now follows by noting that $|E_{n}|=2n(n-1)$. ∎

10 Appendix

10.1 Some auxiliary results

Lemma 10.1**.**

Suppose $\{f_{i},g_{i},h_{i}\}_{i=1}^{n}$ are non negative real numbers satisfying the following inequality for each $i\in[n],$

[TABLE]

Let $\{w_{i}\}_{i=1}^{m}$ be some other non negative numbers. In addition, also suppose the following inequality holds for some $\delta>0,$

[TABLE]

Then the following is true:

[TABLE]

where $a_{+}=\max\{a,0\}$ for any $a\in\mathbb{R}$.

Proof.

The first equation in the above proposition basically says $(f_{i}-g_{i})_{-}\leq h_{i}$ for $i\in[n]$ where $a_{-}=(-a)_{+}$ for any $a\in\mathbb{R}$. Therefore we can write

[TABLE]

which finishes the proof of the lemma. ∎

We state the following lemma which appears as Lemma $D.1$ in guntuboyina2017adaptive.

Lemma 10.2** (Guntuboyina et al.).**

Suppose $p,n\geq 1$ and let $\Theta_{1},\dots,\Theta_{p}$ be subsets of $\mathbb{R}^{n}$ each containing the origin and contained in the closed Euclidean ball of radius $D>0$ centered at the origin. Then for $Z\sim N(0,\sigma^{2}I)$ we have

[TABLE]

Recall that for a vector $v\in\mathbb{R}^{n}$ we define

[TABLE]

Lemma 10.3**.**

Let $\theta\in\mathbb{R}^{n}.$ Let us define $\overline{\theta}=(\sum_{i=1}^{n}\theta_{i})/n.$ Then we have the following inequality:

[TABLE]

Proof.

Define $\alpha_{1}=\theta_{1},\beta_{1}=0$ and for every $i>2$ define

[TABLE]

Now define $\beta=\alpha-\theta.$ Observe that as defined, $\alpha,\beta$ are monotonically non decreasing vectors. Also, we have the equality

[TABLE]

Now we can expand:

[TABLE]

thus giving us the lemma. ∎

Lemma 10.4**.**

Let $\alpha\in\mathbb{R}^{n}$ and let $B_{1},B_{2},\dots,B_{k}$ be a partition of $[n]$ into contiguous blocks. Let $\alpha_{B_{j}}$ denote the restriction of $\alpha$ to the block $B_{j}.$ Also let $\widetilde{\alpha}\in\mathbb{R}^{n}$ be defined so that

[TABLE]

In other words, $\widetilde{\alpha}$ is the best Euclidean approximation to $\alpha$ within the subspace of all vectors which are constant on each block $B_{j}.$ We then have the following inequality:

[TABLE]

Proof.

For any set of indices $i_{1}\in B_{1},\dots,i_{k}\in B_{k},$ we have the following inequality:

[TABLE]

Now averaging over the indices $i_{j}\in B_{j}$ and using Jensen’s inequality gives us

[TABLE]

The last two displays finish the proof of the proposition. ∎

10.2 Proof of Lemma 8.14

Proof.

Let $P_{0}=[n]$ be the initial partition. At every step we take the blocks $b_{i}\in P_{i}$ for which $T(b_{i})>\epsilon$ and divide $b_{i}$ into two equal parts. Let $n_{i}$ be the number of blocks of the partition $P_{i}$ and $s_{i}$ equal the number of blocks $B_{i}$ in $P_{i}$ that are divided to obtain $P_{i+1}.$ Define $s_{0}=0.$ Therefore we have $n_{i+1}=n_{i}+s_{i}.$ Note that, due to superadditivity of $T,$ we must have $s_{i}\leq\lceil\frac{t}{\epsilon}\rceil.$ This implies in particular that $n_{i}\leq 1+i\lceil\frac{t}{\epsilon}\rceil.$ Now the division scheme can go on for atmost $N=\lceil\log_{2}n\rceil$ rounds. Therefore we have

[TABLE]