Concentration inequalities for random tensors

TL;DR

This paper extends concentration inequalities to random tensors with independent entries, demonstrating optimal dimension dependence and showing that most such tensors are well-conditioned and far from linear dependence.

Contribution

It introduces new concentration inequalities for random tensors with optimal dimension and degree dependence, and proves their well-conditioning properties.

Findings

Random tensors satisfy concentration inequalities with optimal dependence on dimension and degree.

Most random tensors are far from linear dependence with high probability.

The results hold for degrees up to o(√(n/ log n)) and conjecture to extend to degrees O(n).

Abstract

We show how to extend several basic concentration inequalities for simple random tensors $X = x_1 \otimes \cdots \otimes x_d$ where all $x_k$ are independent random vectors in $\mathbb{R}^n$ with independent coefficients. The new results have optimal dependence on the dimension $n$ and the degree $d$. As an application, we show that random tensors are well conditioned: $(1-o(1)) n^d$ independent copies of the simple random tensor $X \in \mathbb{R}^{n^d}$ are far from being linearly dependent with high probability. We prove this fact for any degree $d = o(\sqrt{n/\log n})$ and conjecture that it is true for any $d = O(n)$.

Equations

$$\mathbb{P}\left\{\mathinner{\!\left\lvert f(x)-\operatorname{\mathbb{E}}f(x)\right\rvert}>t\rule{0.0pt}{8.53581pt}\right\}\leq 2\exp\Big{(}-\frac{t^{2}}{2\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}^{2}}\Big{)}$$

$$\mathbb{P}\left\{\mathinner{\!\left\lvert f(x)-\operatorname{\mathbb{E}}f(x)\right\rvert}>t\rule{0.0pt}{8.53581pt}\right\}\leq 2\exp\Big{(}-\frac{ct^{2}}{\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}^{2}}\Big{)}.$$

$$\mathbb{P}\left\{\mathinner{\!\left\lvert\mathinner{\!\left\lVert Ax\right\rVert}_{H}-\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{HS}}\right\rvert}\geq t\rule{0.0pt}{8.53581pt}\right\}\leq 2\exp\Big{(}-\frac{ct^{2}}{\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}^{2}}\Big{)}.$$

$$X:=x_1\otimes \cdots \otimes x_d$$

$$E{\Vert x\Vert }_2^2=n\text{and}E{\Vert x\Vert }_2\le \sqrt{n-c}$$

$$\mathrm{Var}(f(X))$$

$$\ge n^d-(n-c{)}^d\ge cd(n-c{)}^{d-1}\text{(by the binomial expansion)}$$

$$\asymp d{n}^{d-1}.$$

$$\mathbb{P}\left\{\mathinner{\!\left\lvert f(X)-\operatorname{\mathbb{E}}f(X)\right\rvert}>t\rule{0.0pt}{8.53581pt}\right\}\leq 2\exp\Big{(}-\frac{ct^{2}}{dn^{d-1}\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}^{2}}\Big{)}.$$

$$\mathbb{P}\left\{\mathinner{\!\left\lvert\mathinner{\!\left\lVert AX\right\rVert}_{H}-\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{HS}}\right\rvert}\geq t\rule{0.0pt}{8.53581pt}\right\}\leq 2\exp\Big{(}-\frac{ct^{2}}{dn^{d-1}\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}^{2}}\Big{)}.$$

$$\mathbb{P}\left\{\left|⟨B,X⟩\right|\ge t\right\}\le C_d\exp \left(-c_d\cdot \underset{1\le k\le d}{\min }\underset{I_1\bigsqcup \cdots \bigsqcup I_k=[d]}{\min }{\left(\frac{t}{{\Vert B\Vert }_{I_1,\dots ,I_k}}\right)}^{2/k}\right).$$

$$\Big{\|}\sum_{i=1}^{m}a_{i}X_{i}\Big{\|}_{2}\geq\sigma\mathinner{\!\left\lVert a\right\rVert}_{2}\quad\text{for all }a=(a_{1},\ldots,a_{m})\in\mathbb{R}^{m}$$

$$\sigma_{\min}\big{(}\mathbf{X}^{\odot d}\big{)}\geq\sigma.$$

$$\sigma_{\min}\big{(}\mathbf{X}^{\odot 1}\big{)}\gtrsim\varepsilon\sqrt{n}$$

$$\Big{\|}\sum_{i=1}^{m}a_{i}X_{i}\Big{\|}_{2}\geq\frac{\sqrt{\varepsilon}}{2}\,\mathinner{\!\left\lVert a\right\rVert}_{2}\quad\text{for all }a=(a_{1},\ldots,a_{m})\in\mathbb{R}^{m}.$$

$$f-Ef=(f-E_{x}f)+(E_{x}f-E_{x,y}f)+(E_{x,y}f-E_{x,y,z}f)=:{\Delta }_1+{\Delta }_2+{\Delta }_3$$

$${\Vert y\Vert }_2{\Vert z\Vert }_2\le K^{d/2}$$

$$f^2-E{f}^2=(f^2-E_x{f}^2)+(E_x{f}^2-E_{x,y}{f}^2)+(E_{x,y}{f}^2-E_{x,y,z}{f}^2)=:{\Delta }_1+{\Delta }_2+{\Delta }_3.$$

$$\mathbb{P}\left\{\left|{\Vert x\Vert }_2-\sqrt{n}\right|>t\right\}\le 2\exp (-c{t}^2).$$

$$\operatorname{\mathbb{E}}\exp\Big{(}\lambda(x^{\mathsf{T}}Mx-\operatorname{tr}M)\Big{)}\leq\exp\left(C\lambda^{2}\mathinner{\!\left\lVert M\right\rVert}_{\mathsf{HS}}^{2}\right)$$

$$S:=x^{\mathsf{T}}Mx-\mathrm{tr}M=\sum _{i=1}^n{M}_{ii}(x_i^2-1)+\sum _{i,j\text{mathchar}58i\ne j}{M}_{ij}{x}_i{x}_j=:S_{\text{diag}}+S_{\text{offdiag}}.$$

$$\operatorname{\mathbb{E}}\exp(\lambda S)\leq\Big{[}\operatorname{\mathbb{E}}\exp\left(2\lambda S_{\textrm{diag}}\right)\Big{]}^{1/2}\Big{[}\operatorname{\mathbb{E}}\exp\left(2\lambda S_{\textrm{offdiag}}\right)\Big{]}^{1/2}.$$

$$\mathinner{\!\bigl{\lVert}x_{i}^{2}-1\bigr{\rVert}}_{\psi_{1}}\lesssim\mathinner{\!\bigl{\lVert}x_{i}^{2}\bigr{\rVert}}_{\psi_{1}}=\mathinner{\!\bigl{\lVert}x_{i}\bigr{\rVert}}_{\psi_{2}}^{2}\lesssim 1.$$

$$E\exp \left({\lambda }_i(x_i^2-1)\right)\le \exp (C{\lambda }_i^2)\text{if }\left|{\lambda }_i\right|\le c.$$

$$E\exp (2\lambda S_{\text{diag}})$$

$$\displaystyle\leq\exp\Big{(}C\lambda^{2}\sum_{i=1}^{n}M_{ii}^{2}\Big{)}\quad\text{if }\mathinner{\!\left\lvert\lambda\right\rvert}\leq\frac{c}{\mathinner{\!\left\lvert M_{ii}\right\rvert}}\quad\text{(by \eqref{eq: MGF xi2})}$$

$$\displaystyle\leq\exp\Big{(}C\lambda^{2}\mathinner{\!\left\lVert M\right\rVert}_{\mathsf{HS}}^{2}\Big{)}\quad\text{if }\mathinner{\!\left\lvert\lambda\right\rvert}\leq\frac{c}{\mathinner{\!\left\lVert M\right\rVert}_{\mathsf{op}}}.$$

$$E\exp (2\lambda S_{\text{offdiag}})$$

$$\displaystyle\leq\operatorname{\mathbb{E}}\exp\Big{(}8\lambda\sum_{i,j=1}^{n}M_{ij}x_{i}x_{j}^{\prime}\Big{)}\quad\text{(by decoupling, see \cite[cite]{[\@@bibref{}{V book}{}{}, Remark~{}6.1.3]})}$$

$$\displaystyle\leq\exp\Big{(}C\lambda^{2}\mathinner{\!\left\lVert M\right\rVert}_{\mathsf{HS}}^{2}\Big{)}\quad\text{if }\mathinner{\!\left\lvert\lambda\right\rvert}\leq\frac{c}{\mathinner{\!\left\lVert M\right\rVert}_{\mathsf{op}}},$$

$$\operatorname{\mathbb{E}}\exp\Big{(}\lambda\big{(}\mathinner{\!\left\lVert Ax\right\rVert}_{H}^{2}-\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{HS}}^{2}\big{)}\Big{)}\leq\exp\left(C\lambda^{2}\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}^{2}\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{HS}}^{2}\right)$$

$$x^{\mathsf{T}}Mx={\Vert Ax\Vert }_H^2,\mathrm{tr}M={\Vert A\Vert }_{\mathsf{HS}}^2,{\Vert M\Vert }_{\mathsf{op}}={\Vert A\Vert }_{\mathsf{op}}^2,{\Vert M\Vert }_{\mathsf{HS}}\le {\Vert A\Vert }_{\mathsf{op}}{\Vert A\Vert }_{\mathsf{HS}}.\text{qed}$$

Full text

Work in part supported by the U.S. Air Force grant FA9550-18-1-0031.

1. Introduction

Concentration inequalities form a powerful toolset in probability theory and its many applications; see e.g. [12, 18, 19]. Perhaps the best known member of this large family of results is the Gaussian concentration inequality, which states that a standard normal random vector $x$ in $\mathbb{R}^{n}$ satisfies

[TABLE]

for any Lipschitz function $f\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to\mathbb{R}$, see e.g. [12, Theorem 5.6].

The Gaussian concentration inequality can be extended to some more general distributions on $\mathbb{R}^{n}$. A remarkable situation where this is possible is where $x$ has a product distribution with bounded coordinates and $f$ is convex. This result is due to by M. Talagrand [28]; see [18, Section 4.2], [12, Section 7.5]:

Theorem 1.1** (Convex concentration).**

Let $f\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to\mathbb{R}$ be a convex and Lipschitz function. Let $x$ be a random vector in $\mathbb{R}^{n}$ whose coordinates are independent random variables that are bounded a.s. Then, for every $t\geq 0$, we have

[TABLE]

Here $c>0$ depends only on the bound on the coordinates.

The boundedness assumption in this result unfortunately excludes Gaussian distributions and many others. Significant efforts were made to extend Gaussian concentration to more general, not necessarily bounded, distributions, see e.g. [2, 6, 15] and the references therein. One such result, which holds for a general random vector $x$ with independent subgaussian coordinates, is for Euclidean functions, i.e. the functions of the form $f(x)=\mathinner{\!\left\lVert Ax\right\rVert}_{H}$ where $A$ is a linear operator from $\mathbb{R}^{n}$ into a Hilbert space.

Theorem 1.2** (Euclidean concentration).**

Let $H$ be a Hilbert space and $A\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to H$ be a linear operator. Let $x$ be a random vector in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then, for every $t\geq 0$, we have

[TABLE]

Here $c>0$ depends only on the bound on the subgaussian norms.

In this result, $\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{HS}}$ and $\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}$ denote the Hilbert-Schmidt and operator norms of $A$, respectively. Theorem 1.2 can be derived from Hanson-Wright concentration inequality for quadratic forms [27], see [35, Section 6.3].

1.1. New results

The goal of this paper is to demonstrate how Theorems 1.1 and 1.2 can be extended for simple random tensors. These are tensors of the form

[TABLE]

where $x_{k}$ are independent random vectors in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance random variables that are either bounded a.s. (in Theorem 1.1) or subgaussian (in Theorem 1.2).

Can we expect that a concentration inequality like (1.2) or (1.3) can hold for simple random tensors, i.e. for $f(X)$ where $f\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n^{d}},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to\mathbb{R}$? Not really: if such inequality did hold, then it would imply that $\operatorname{Var}(f(X))=O(1)$, but this is not the case. Indeed, consider the simplest case where $f$ is given by the Euclidean norm, i.e. $f(X)\coloneqq\mathinner{\!\left\lVert X\right\rVert}_{2}$, and let all $x_{k}$ be standard normal random vectors in $\mathbb{R}^{n}$. Recall that a standard normal vector $x$ in $\mathbb{R}^{n}$ satisfies

[TABLE]

for $n$ large enough, where $c>0$ is an absolute constant.111To check the second bound, write $\mathinner{\!\left\lVert x\right\rVert}_{2}^{2}=n+\sum_{i=1}^{n}(x_{i}^{2}-1)$ and observe that, by the central limit theorem, the sum is approximately $\sqrt{2n}\,g$ where $g\sim N(0,1)$. Thus $\mathinner{\!\left\lVert x\right\rVert}_{2}\approx\sqrt{n}+g/\sqrt{2}$, so $\operatorname{Var}(\mathinner{\!\left\lVert x\right\rVert}_{2})\gtrsim c$. On the other hand, $\operatorname{Var}(\mathinner{\!\left\lVert x\right\rVert}_{2})=\operatorname{\mathbb{E}}\mathinner{\!\left\lVert x\right\rVert}^{2}-(\operatorname{\mathbb{E}}\mathinner{\!\left\lVert x\right\rVert})^{2}=n-(\operatorname{\mathbb{E}}\mathinner{\!\left\lVert x\right\rVert})^{2}$. Thus, $\operatorname{\mathbb{E}}\mathinner{\!\left\lVert x\right\rVert}\leq\sqrt{n-c}$. Then

[TABLE]

Thus, the strongest concentration inequality we can hope for must have the form

[TABLE]

Such inequality, however, can not be true for large $t$. The coefficients of $X$ are $d$-fold products of subgaussian random variables, and such products for $d\geq 2$ typically have tails that are heavier than subgaussian. Nevertheless, we may still hope that the inequality (1.4) might hold for all $t$ in some interesting range, for example for $0\leq t\leq\mathinner{\!\left\lvert\operatorname{\mathbb{E}}f(X)\right\rvert}$. This is what we prove in the current paper.

Theorem 1.3** (Convex concentration for random tensors).**

Let $n$ and $d$ be positive integers and $f\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n^{d}},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to\mathbb{R}$ be a convex and Lipschitz function. Consider a simple random tensor $X\coloneqq x_{1}\otimes\cdots\otimes x_{d}$ in $\mathbb{R}^{n^{d}}$, where all $x_{k}$ are independent random vectors in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance random variables that are bounded a.s. Then, for every 0\leq t\leq 2\big{(}\operatorname{\mathbb{E}}\mathinner{\lvert f(X)\rvert}^{2}\big{)}^{1/2}, we have

[TABLE]

Here $c>0$ depends only on the bound on the coordinates.

Theorem 1.4** (Euclidean concentration for random tensors).**

Let $n$ and $d$ be positive integers, $H$ be a Hilbert space and $A\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n^{d}},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to H$ be a linear operator. Consider a simple random tensor $X\coloneqq x_{1}\otimes\cdots\otimes x_{d}$ in $\mathbb{R}^{n^{d}}$, where all $x_{k}$ are independent random vectors in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then, for every $0\leq t\leq 2\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{HS}}$, we have

[TABLE]

Here $c>0$ depends only on the bound on the subgaussian norms.

Remark 1.5* (The range of concentration inequalities).*

Our argument shows that the main results actually hold in a somewhat wider range of $t$, namely for $0\leq t\leq 2n^{d/2}\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}$ in Theorem 1.3 and $0\leq t\leq 2n^{d/2}\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}$ in Theorem 1.4.

1.2. Related results

Several existing techniques are already known to shed light on tensor concentration. Indeed, the quantity $\mathinner{\!\left\lVert AX\right\rVert}_{H}^{2}$ in Theorem 1.4 can be expressed a polynomial of degree $2d$ in $nd$ independent subgaussian random variables, which are the coefficients of the random vectors $x_{i}$. A remarkable result of Latala [17] provides two-sided bounds on the moments of polynomials of independent normal random variables, or Gaussian chaoses. These moment bounds can be translated into the following concentration inequality that is valid for any fixed tensor $B\in\mathbb{R}^{n^{d}}$ and a simple random tensor $X\coloneqq x_{1}\otimes\cdots\otimes x_{d}$ composed of independent, mean zero, standard normal random vectors $x_{k}$ in $\mathbb{R}^{n}$:

[TABLE]

The second minimum in the right hand side is over all partitions of the index set $[d]=\{1,\ldots,d\}$ into $k$ subsets, and $\mathinner{\!\left\lVert B\right\rVert}_{I_{1},\ldots,I_{k}}$ are certain norms of $B$, which interpolate between the Hilbert-Schmidt and the operator norms; see [17] for details.

Lehec [20] gave an alternative, shorter proof of (1.5) based on Talagrand’s majorizing measure theorem. Adamczak and Latala [4] extended (1.5) for log-concave distributions. Adamczak and Wolff [7] considered distributions that satisfy a Sobolev inequality, and he gave an extension of (1.5) for general differentiable functions of $X$ (not just polynomials). Götze, Sambale, and Sinulis extended (1.5) for $\alpha$-subexponential distributions. Very recently, Adamczak, Latala, and Meller [5] explored extensions of (1.5) where the coefficients of $B$ are not scalars but vectors in some Banach space. We refer the paper to the papers [7, 5] for a review of the vast literature on concentration inequalities for multivariate polynomials, U-statistics, and more general functions of independent random variables.

As we noted, inequality (1.5) is well suited to study concentration properties of the quantity $\mathinner{\!\left\lVert AX\right\rVert}_{H}^{2}$ that appears in Theorem 1.4. However, Latala’s concentration inequality (1.5), as well as all its known extensions and ramifications, contain factors $C_{d}$ and $c_{d}$ that depend on the degree $d$ of tensors in some unspecified way; the dependence seems to be at least exponential in $d$. In contrast, Theorems 1.3 and 1.4 feature the optimal dependence on the degree $d$ of tensors: the constant $c$ in both theorems does not depend on $d$ at all. This can be critical in some applications, one of which we discuss next.

1.3. Application: random tensors are well conditioned

Our work was primarily inspired by a question that arose recently in the theoretical computer science community [11, 9, 1, 8]: Are random tensors well conditioned?

Suppose $X_{1},\ldots,X_{m}$ are independent copies of a simple random tensor $X\coloneqq x_{1}\otimes\cdots\otimes x_{d}$. How large can $m$ be so that these random tensors are linearly independent with high probability? Certainly $m$ can not exceed the dimension $n^{d}$ of the tensor space, but can it be arbitrarily close to the dimension, say $m=0.99n^{d}$? Moreover, instead of linear independence we may ask for a stronger, more quantitative property of being well conditioned. We would like to have a uniform bound

[TABLE]

with $\sigma$ as large as possible. Equivalently, we can understand (1.6) as a lower bound on the smallest singular value of the $n^{d}\times m$ matrix $\mathbf{X}^{\odot d}$ (called the Khatri-Sidak product) whose columns are vectorized tensors $X_{1},\ldots,X_{m}$:

[TABLE]

Problems of this type were studied recently in the theoretical computer science community in the context of computing tensor decompositions [11, 8], learning Gaussian mixtures [9] and estimating the capacity of error correcting codes [1].

For $d=1$, this problem has been extensively studied in random matrix theory [16, 30, 22, 31, 24, 3, 25, 32, 26, 13, 10, 33, 34]. Since $\mathbf{X}^{\odot 1}$ is an $n\times m$ random matrix whose entries are independent, mean zero, unit variance, subgaussian random variables, known optimal results [25] yield the bound

[TABLE]

if $m=(1-\varepsilon)n$ and $\varepsilon\in(0,1]$.

For $d\geq 2$, optimal results on \sigma=\sigma_{\min}\big{(}\mathbf{X}^{\odot d}\big{)} are yet unknown. Various random models were studied: the factors $x_{k}$ of the simple tensor $X$ were assumed to be Gaussian (possibly with nonzero means) in [11], Bernoulli in [1] and are allowed to have very general general distributions with non-degenerate marginals in [8]. Symmetric random tensors are considered in [1]. Baskara et al. [11] obtained the lower bound (1.6) with $\sigma=(1/n)^{\exp(O(d))}$ for $d=o(\log\log n)$; Anari et al. [8] improves this to $\sigma=(1/O(n))^{d}$ for $d=o(\sqrt{n/\log n})$, and Abbe et al. [1] guarantees linear independence (i.e. $\sigma>0$) for symmetric random tensors if $d=o(\sqrt{n/\log n})$. For a related notion of row product of random matrices, the problem was studied by Rudelson [23].

In this paper, we prove a bound on the smallest singular value \sigma_{\min}\big{(}\mathbf{X}^{\odot d}\big{)} is of constant order. We derive it as an application of Theorem 1.4. Let us give an informal statement here; Corollary 6.2 will provide a more rigorous version.

Corollary 1.6** (Random tensors are well conditioned).**

If $d=o(\sqrt{n/\log n})$ and $\varepsilon\in(0,1)$, then $m=(1-\varepsilon)n^{d}$ independent simple subgaussian random tensors $X_{1},\ldots,X_{m}$ in $\mathbb{R}^{n^{d}}$ are well conditioned with high probability:

[TABLE]

1.4. Our approach

Let us briefly explain our approach to tensor concentration. Suppose first that we do not care about the dependence on the degree $d$. Then Theorem 1.3 can be proved by expressing the deviation $f-\operatorname{\mathbb{E}}f$ as a telescopic sum and controlling each increment by Talagrand’s Theorem 1.1. For example, if $d=3$, then for the function $f=f(x\otimes y\otimes z)$ we would write

[TABLE]

where $\operatorname{\mathbb{E}}_{x}$ denotes the conditional expectation with respect to $x$ (conditional on $y$ and $z$), and similarly for $\operatorname{\mathbb{E}}_{x,y}$. Applying Theorem 1.1 for $f$ as a function of $x$, we control $\Delta_{1}$; applying the same theorem for $\operatorname{\mathbb{E}}_{x}f$ as a function of $y$, we control $\Delta_{2}$, and applying it again for $\operatorname{\mathbb{E}}_{x,y}f$ as a function of $z$, we control $\Delta_{3}$. Then we combine all increments $\Delta_{k}$ by the triangle inequality.

This argument, however, would produce an exponential dependence of $d$ in the concentration inequality. This is because the Lipschitz norm of $f$ as a function of $x$ is bounded by

[TABLE]

if all coefficients of $y$ and $z$ are bounded by $K$ a.s.

To get a better control of the Lipschitz norms of all functions that appear. in the telescopic sum, we prove a maximal inequality in Section 3, which provides us with a uniform bound on the products of norms of independent random vectors. This allows us to avoid losing any factors that are exponential in $d$.

However, combining the increments $\Delta_{k}$ by a simple union bound and triangle inequality is suboptimal and leads to an extra factor that is linear in $d$. One can avoid this by noting that $\Delta_{k}$ are martingale differences and using martingale concentration techniques (coupled with the maximal inequality). This is the approach we chose to prove Theorem 1.3.

One can try to prove Theorem 1.4 in a similar way, but a new difficulty arises here. We may not simply choose $f(x\otimes y\otimes z)=\mathinner{\!\left\lVert A(x\otimes y\otimes z)\right\rVert}_{2}$ and write the telescopic sum for it. This is because $\operatorname{\mathbb{E}}_{x}f$ is not a Euclidean function in $y$, i.e. it can not be expressed as $\mathinner{\!\left\lVert By\right\rVert}$ for any linear operator $B$ mapping $\mathbb{R}^{n}$ into a Hilbert space, so we may not use Theorem 1.2 to control the deviation of $\operatorname{\mathbb{E}}_{x}f$.

This forces us to work with $f^{2}$ instead of $f$, since then $(\operatorname{\mathbb{E}}_{x}f^{2})^{1/2}$ is a Euclidean function. Thus we write

[TABLE]

Squaring $f$, however, produces tails of the increments that are heavier than subgaussian. This prompts us to abandon the use of Theorem 1.2. Instead of controlling the tails of the increments, we control their moment generating function (MGF). In the end, we still combine the MGF’s of the increments using a martingale-like argument coupled with the maximal inequality. This ultimately leads to Theorem 1.4.

Remark 1.7* (An alternative approach to convex concentration for random tensors).*

There is an alternative and somewhat simpler way to prove Theorem 1.3, where one won’t have to use a maximal inequality. Instead, one can deduce this result (with some work) from a version of convex concentration (Theorem 1.1) that holds for a weaker notion of convexity, namely for separately convex functions, i.e. functions that are convex in each coordinate [28, 21], see [12, Theorems 6.10, 6.9]. However, there seem to be no simpler way to prove Theorem 1.4, a result we care most about in view of applications. So, for pedagogical reasons we choose to prove Theorem 1.3 using maximal inequality, so we can use it later as a stepping stone for the proof of Theorem 1.4.

Remark 1.8* (A broader view).*

The method we develop here is flexible and might be used to “tensorize” some other concentration inequalities. For example, if all $x_{k}$ have the standard normal distribution, then the convexity requirement is not needed in Theorem 1.3, and we get a tensor version of the Gaussian concentration inequality (1.1). Furthermore, one should be able to relax the subgaussian assumption in Theorem 1.4 by using in our argument a version Theorem 1.2 for heavier tails obtained recently by Götze, Sambale, and Sinulis [14].

1.5. Open problems

We do not know optimal concentration inequalities for symmetric tensors $X=x^{\otimes d}=x\otimes\cdots\otimes x$. One could possibly use decoupling to reduce the problem to concentration to tensors with independent factors, and then apply Theorem 1.3 or 1.4. However, decoupling will likely cause a loss of factors that are exponential in $d$, which will defeat our purpose.

There are many directions in which Corollary 1.6 should be strengthened and generalized. Can the bound be improved to $\varepsilon n^{d/2}$, matching the inequality (1.7) for $d=1$? Does it hold for degrees higher than $\sqrt{n}$, for example for $d\asymp n$? Even linear independence is unknown for higher degrees. Can Corollary 1.6 be extended to other models of randomness considered in the theoretical computer science community [11, 9, 1, 8]? For example, does it hold for symmetric tensors, and can the mean zero and subgaussian assumptions be significantly weakened?

1.6. The rest of the paper

In Section 2 we collect some basic facts from high-dimensional probability that will be needed later. Most importantly, in Proposition 2.2 we show how to control the MGF of a random chaos of order $2$, which is the quadratic form $x^{\mathsf{T}}Mx$ where $x$ is a random vector and $M$ is a fixed matrix. In Proposition 2.2 we derive a version of Hanson-Wright inequality in the MGF form. These results, although possibly known as a folklore, are hard to find in the literature and could be useful for future applications.

In Section 3 we prove a sharp maximal inequality for products of norms of independent random vectors. We use it to establish our main results: in Section 4 we prove Theorem 1.3 and in Section 5 we prove Theorem 1.4.

In Section 6 we give two applications to the geometry of random tensors. We prove a concentration inequality for the distance between a random tensor and a given subspace in Corollary 6.1, and then use it to show that random tensors are well conditioned, proving a formal version of Corollary 1.6.

Acknowledgement

The author is grateful to Mark Rudelson for useful comments on the preliminary version of this manuscript, in particular for pointing out an alternative approach to Theorem 1.1 mentioned in Remark 1.7.

2. Preliminaries

Throughout this paper, we use basic facts about subgaussian and subexponential random variables that can be found e.g. in [35, Chapters 2–3] and [36, Chapter 2]. Positive constants are denoted by $C,c,C_{1},c_{1},\ldots$, and their specific values can be different in different parts of this paper. We allow these constants to depend only on the a.s. bound of the coefficients (for Theorem 1.1) or the subgaussian norms of the coefficients (for Theorem 1.2), but not on any other parameters.

2.1. Concentration of the norm

One such fact that follows immediately from Theorem 1.2 for the identity matrix $A$ is a concentration inequality for the norm of a random vector.

Corollary 2.1** (Concentration of norm).**

Let $x$ be a random vector in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then, for every $t\geq 0$, we have

[TABLE]

Here $c>0$ depends only on the bound on the subgaussian norms.

2.2. MGF of a quadratic form

Another useful fact is the following bound on the moment generating function (MGF) of a chaos of order $2$. It might be known but is hard to find in the literature in this generality.

Proposition 2.2** (MGF of a subgaussian chaos).**

Let $M$ be an $n\times n$ matrix. Let $x$ be a random vector in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then

[TABLE]

for every $\lambda\in\mathbb{R}$ such that $\mathinner{\!\left\lvert\lambda\right\rvert}\leq c/\mathinner{\!\left\lVert M\right\rVert}_{\mathsf{op}}$.

Proof.

Step 1. Separating the diagonal and off-diagonal parts. We can break the quadratic form as follows:

[TABLE]

By Cauchy-Schwarz inequality, we have

[TABLE]

Let us consider the diagonal and off-diagonal parts separately.

Step 2. Diagonal part. Since $x_{i}$ are subgaussian random variables with unit variance, $x_{i}^{2}-1$ are mean-zero, subexponential random variables, and

[TABLE]

(This is a combination of some basic facts about subgaussian and subexponential distributions, see [35, Exercise 2.7.10 and Lemma 2.7.6].) Then a standard bound on the MGF of a mean-zero, subexponential distribution ([35, Property 5 in Proposition 2.7.1]) gives

[TABLE]

Therefore

[TABLE]

Step 2. Off-diagonal part. Let $x_{1}^{\prime},\ldots,x_{n}^{\prime}$ be independent copies of $x_{1},\ldots,x_{n}$. We have

[TABLE]

where the last bound follows from [35, Lemmas 6.2.3 and 6.2.2].

Combining the diagonal and off-diagonal contributions, we complete the proof. ∎

Corollary 2.3**.**

Let $H$ be a Hilbert space and $A\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to H$ be a linear operator. Let $x$ be a random vector in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then

[TABLE]

for every $\lambda\in\mathbb{R}$ such that $\mathinner{\!\left\lvert\lambda\right\rvert}\leq c/\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}^{2}$.

Proof.

Apply Proposition 2.2 for $M\coloneqq A^{*}A$ and note that

[TABLE]

Note in passing that Corollary 2.3 implies Euclidean concentration Theorem 1.2. All one needs to do is use exponential Markov’s inequality and optimize the resulting bound in $\lambda$. We leave this as an exercise.

2.3. Euclidean functions

Theorems 1.2 and 1.4 can be conveniently stated as results about concentration of Euclidean functions.

Definition 2.4** (Euclidean functions).**

A function $f\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n}\to[0,\infty)$ is called a Euclidean function on $\mathbb{R}^{n}$ if it can be expressed as

[TABLE]

where $H$ is a Hilbert space and $A\mathrel{\mathop{\mathchar 58\relax}}\mathbb{R}^{n}\to H$ is a linear operator. Equivalently, $f$ is Euclidean if $f^{2}$ is a positive-semidefinite quadratic form, i.e. if

[TABLE]

for some $n\times n$ positive-semidefinite matrix $M$.

Let us note a few obvious facts about Euclidean functions:

Lemma 2.5** (Properties of Euclidean functions).**

1. (i)

If $f$ is a Euclidean function on $\mathbb{R}^{n}$ then $af$ is, for any $a\geq 0$. 2. (ii)

If $f$ and $g$ are Euclidean functions on $\mathbb{R}^{n}$ then $\sqrt{f^{2}+g^{2}}$ is. 3. (iii)

If $f$ is a random Euclidean function on $\mathbb{R}^{n}$ then $(\operatorname{\mathbb{E}}f^{2})^{1/2}$ is. 4. (iv)

The Lipschitz norm of a Euclidean function can be computed as follows:

[TABLE]

In particular, if $f(x)=\mathinner{\!\left\lVert Ax\right\rVert}_{H}$ then $\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}=\mathinner{\!\left\lVert A\right\rVert}_{\mathsf{op}}$.

For future convenience, we restate Corollary 2.3 in terms of Euclidean functions:

Corollary 2.6** (MGF of a Euclidean function).**

Let $f\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to[0,\infty)$ be a Euclidean function. Let $x$ be a random vector in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then

[TABLE]

for every $\lambda\in\mathbb{R}$ such that $\mathinner{\!\left\lvert\lambda\right\rvert}\leq c/\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}^{2}$.

3. A maximal inequality

The proof of both of our main results, Theorems 1.3 and 1.4, relies on a tight control of the norm of the simple random tensor

[TABLE]

Lemma 3.1** (The norm of a random tensor).**

Let $x_{1},\ldots,x_{d}\in\mathbb{R}^{n}$ be independent random vectors with independent, mean zero, unit variance, subgaussian coordinates. Then, for every $0\leq t\leq 2n^{d/2}$, we have

[TABLE]

Note in passing that this result is a partial case of Theorem 1.3 for the function $f(X)=\mathinner{\!\left\lVert X\right\rVert}_{2}$ and of Theorem 1.4 for the identity map $A$.

Proof.

Let $s\geq 0$. Then

[TABLE]

To check this, take $d$-th root on both sides of the inequality \prod_{i=1}^{d}\mathinner{\!\bigl{\lVert}x_{i}\bigr{\rVert}}_{2}>(\sqrt{n}+s)^{d}, apply the inequality of arithmetic and geometric means, and subtract $\sqrt{n}$ from both parts. Furthermore, we can replace all terms $\sqrt{n}$ in (3.2) by the means \mu_{i}\coloneqq\operatorname{\mathbb{E}}\mathinner{\!\bigl{\lVert}x_{i}\bigr{\rVert}}_{2} since they satisfy

[TABLE]

Thus the probability in (3.2) is bounded by

[TABLE]

which is a tail probability for a sum of independent, mean zero random variables.

By the concentration of the norm (Corollary 2.1) and a standard centering argument ([35, Lemma 2.6.8]), we have222Here $\mathinner{\!\left\lVert\cdot\right\rVert}_{\psi_{2}}$ and $\mathinner{\!\left\lVert\cdot\right\rVert}_{\psi_{1}}$ denote the subgaussian and subexponential norms, respectively; see [35, Sections 2.5, 2.7] for definition and basic properties.

[TABLE]

and this implies that

[TABLE]

see [35, Proposition 2.6.1]. Thus the probability in (3.3) is bounded by

[TABLE]

Let $0\leq u\leq 2$ and apply this bound for $s\coloneqq u\sqrt{n}/(2d)$. With this choice,

[TABLE]

Thus we have shown that

[TABLE]

Using this inequality for $u\coloneqq t/n^{d/2}$, we complete the proof. ∎

A stronger statement will be needed in the proof of our main results: we will require a tight control of the products (3.4) for all $d$ simultaneously. The following maximal inequality will be used for that:

Lemma 3.2** (A maximal inequality).**

Let $x_{1},\ldots,x_{d}\in\mathbb{R}^{n}$ be independent random vectors with independent, mean zero, unit variance, subgaussian coordinates. Then, for every $0\leq u\leq 2$, we have

[TABLE]

Proof.

Step 1. A binary partition. By increasing $d$ if necessary, we can assume that

[TABLE]

For each level $\ell\in\{0,1,\ldots,L\}$, consider the partition $\mathcal{I}_{\ell}$ of the integer interval $[1,d]=\{1,\ldots,d\}$ into $2^{\ell}$ successive intervals of length

[TABLE]

We call each of these intervals a binary interval. For example, the family $\mathcal{I}_{0}$ consists of just one binary interval $[1,d]$, and the family $\mathcal{I}_{1}$ consists of two binary intervals $[1,d/2]$ and $[d/2+1,d]$.

For every integer $k\in[1,d]$, the interval $[1,k]$ can be partitioned into binary intervals of different lengths. (The binary representation of the number $k/d$ determines which intervals participate in this partition.) As a consequence, such partition of $[1,k]$ must include no more than one interval from each family $\mathcal{I}_{\ell}$.

Step 2. Controlling the product over binary sets. Fix a level $\ell\in\{0,1,\ldots,L\}$ and a binary interval $I\in\mathcal{I}_{\ell}$. Apply Lemma 3.1 with $d$ replaced by $|I|=d_{\ell}=d/2^{\ell}$ and for $t\coloneqq 2^{-\ell/4}n^{d_{\ell}/2}u$. It gives

[TABLE]

Taking a union bound over all levels $\ell$ and all $2^{\ell}$ binary intervals $I$ in the family $\mathcal{I}_{\ell}$, we get

[TABLE]

To simplify this bound, we can assume that $c_{0}nu^{2}/d\geq 1$, otherwise the probability bound in the conclusion of the lemma becomes trivial if $c<c_{0}/2$. Also, $2^{\ell/2}\geq 1$, and thus

[TABLE]

Substituting this into our probability bound, we can continue it as

[TABLE]

By reducing the absolute constant $c_{0}$, we can make $C=2$.

Step 3. Controlling the product over any interval. Let us fix a realization of random vectors for which the good event considered above occurs, i.e.

[TABLE]

Let $1\leq k\leq d$. As we noted in Step 1, we can partition the interval $[1,k]$ into binary intervals $I\in\mathcal{I}_{\ell}$ so that at most one binary interval is taken from each family $\mathcal{I}_{\ell}$. Let us multiply the inequalities (3.5) for all binary intervals $I$ that participate in this partition. Note that sum of exponents $d_{\ell}$ is the sum of the length of these intervals $I$, which equals $k$. Thus we obbtain

[TABLE]

This yields the conclusion of the lemma with $Cu$ instead of $u$ in the bound. One can get rid of $C$ by reducing the constant $c$ in the probability bound. The proof is complete. ∎

4. Proof of Theorem 1.3

Step 1. Applying the maximal inequality. We can assume without loss of generality that $\mathinner{\!\left\lVert f\right\rVert}_{\mathsf{Lip}}=1$. Consider the events

[TABLE]

and let $\mathcal{E}_{d+1}$ be the entire probability space for convenience. Applying the maximal inequality of Lemma 3.2 for $u=1$ and for the reverse ordering of the vectors, we see that the event

[TABLE]

is likely:

[TABLE]

Step 2. Applying the convex concentration inequality. Fix any realization of the random vectors $x_{2},\ldots,x_{d}$ that satisfy $\mathcal{E}_{2}$ and apply the Convex Concentration Theorem 1.1 for $f$ as a function of $x_{1}$. It is a convex and Lipschitz function. To get a quantitative bound on its Lipschitz norm, consider any $x,y\in\mathbb{R}^{n}$ and note that

[TABLE]

This shows that $f$ as a function of $x_{1}$ has Lipschitz norm bounded by

[TABLE]

The Convex Concentration Theorem 1.1 then yields

[TABLE]

In this inequality, $x_{1}$ indicates that the expectation and the $\psi_{2}$ norm is taken with respect to the random vector $x_{1}$, i.e. conditioned on all other random vectors.

Fix any realization of the random vectors $x_{3},\ldots,x_{d}$ that satisfy $\mathcal{E}_{3}$ and apply the Convex Concentration Theorem 1.1 for $\operatorname{\mathbb{E}}_{x_{1}}f$ as a function of $x_{2}$. It is a convex and Lipschitz function. To get a quantitative bound on its Lipschitz norm, consider any $x,y\in\mathbb{R}^{n}$ and note that

[TABLE]

This shows that $\operatorname{\mathbb{E}}_{x_{1}}f$ as a function of $x_{2}$ has Lipschitz norm bounded by $L=2n^{(d-1)/2}$. The Convex Concentration Theorem 1.1 then yields

[TABLE]

Continuing in a similar way, we can show that for every $k=1,\ldots,d$:

[TABLE]

Step 3. Combining the increments using a martingale-like argument. Let us look at the differences

[TABLE]

The estimate (4.3) on the subgaussian norm yields the following bound on the moment generating function [35, Proposition 2.5.2]:

[TABLE]

and for any $\lambda\in\mathbb{R}$. We can combine these pieces using a martingale-like argument, which we defer to Lemma 4.1. It gives that for any $\lambda\in\mathbb{R}$,

[TABLE]

where $\mathcal{E}=\mathcal{E}_{2}\cap\cdots\cap\mathcal{E}_{d}$ is the event whose probability we estimated in (4.1).

Step 4. Deriving the concentration via exponential Markov’s inequality. To derive a probability bound, we can use a standard argument based on exponential Markov’s inequality. Namely, we have for every $\lambda>0$:

[TABLE]

This bound is minimized for $\lambda\coloneqq t/(2CdL^{2})$. With this choice of $\lambda$, and with the choice of $L$ made in (4.2), our bound becomes

[TABLE]

Since by assumption of the theorem,

[TABLE]

we have

[TABLE]

This implies that the first term in the bound (4.5) dominates over the second, if the constant $C$ is sufficiently large compared to $1/c$. This gives

[TABLE]

Finally, repeating the argument for $-f$ instead of $f$ we obtain the same probability bound for $\mathbb{P}\left\{-f+\operatorname{\mathbb{E}}f>t\rule{0.0pt}{8.53581pt}\right\}$. Combining the two bounds, we get

[TABLE]

We can replace the factor $6$ by $2$ by making $C$ larger if necessary. Theorem 1.3 is proved. ∎

Our argument above used on the following martingale-like inequality, which we will carefully state and prove now.

Lemma 4.1** (A martingale-type inequality).**

Let $x_{1},\ldots,x_{d}$ be independent random vectors. For each $k=1,\ldots,d$, let $f_{k}=f_{k}(x_{k},\ldots,x_{d})$ be an integrable real-valued function and $\mathcal{E}_{k}$ be an event that is uniquely determined by the vectors $x_{k+1},\ldots,x_{d}$. Let $\mathcal{E}_{d+1}$ be the entire probability space for convenience. Suppose that, for every $k=1,\ldots,d$:

[TABLE]

Then, for $\mathcal{E}\coloneqq\mathcal{E}_{2}\cap\cdots\cap\mathcal{E}_{d}$, we have

[TABLE]

Proof.

We have

[TABLE]

since $\operatorname{\mathbb{E}}_{x_{1}}\exp(f_{1}){\textbf{1}}_{\mathcal{E}_{2}}\leq\pi_{1}$ a.s. by assumption. Iterating this argument, we complete the proof. ∎

5. Proof of Theorem 1.4

Let us restate Theorem 1.4 in terms of Euclidean functions which were introduced in Section 2.3.

Theorem 5.1** (Euclidean concentration for random tensors).**

Let $n$ and $d$ be positive integers and $f\mathrel{\mathop{\mathchar 58\relax}}(\mathbb{R}^{n^{d}},\mathinner{\!\left\lVert\cdot\right\rVert}_{2})\to[0,\infty)$ be a Euclidean function. Consider a simple random tensor $X\coloneqq x_{1}\otimes\cdots\otimes x_{d}$ in $\mathbb{R}^{n^{d}}$, where all $x_{k}$ are independent random vectors in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then, for every $0\leq t\leq 2(\operatorname{\mathbb{E}}f(X)^{2})^{1/2}$, we have

[TABLE]

Here $c>0$ depends only on the bound on the subgaussian norms.

Proof.

Step 1. Applying the maximal inequality. The proof starts as in the proof of Theorem 1.3 in Section 4. We define the norm-controlling events $\mathcal{E}_{k}$ and estimate the probability of $\mathcal{E}=\mathcal{E}_{2}\cap\cdots\mathcal{E}_{d}$ by a maximal inequality in the same way as before.

Step 2. Applying a subgaussian concentration inequality. Fix any realization of the random vectors $x_{2},\ldots,x_{d}$ that satisfy $\mathcal{E}_{2}$ and apply Corollary 2.6 for $f$ as a function of $x_{1}$. It is a Euclidean function, and one can check as before that its Lipschitz norm is bounded by

[TABLE]

Corollary 2.6 then yields

[TABLE]

provided that $\mathinner{\!\left\lvert\lambda\right\rvert}\leq c/L^{2}$. For future convenience, let us restate this bound as follows. Choose $\lambda\in\mathbb{R}$ and denote

[TABLE]

Then we have

[TABLE]

provided that $\mathinner{\!\left\lvert\lambda_{0}\right\rvert}\leq c/L^{2}$.

Fix any realization of the random vectors $x_{3},\ldots,x_{d}$ that satisfy $\mathcal{E}_{3}$ and apply Corollary 2.6 for $(\operatorname{\mathbb{E}}_{x_{1}}f^{2})^{1/2}$ as a function of $x_{2}$. It is a Euclidean function whose Lipschitz norm is bounded by $L$ as before. Corollary 2.6 then yields

[TABLE]

provided that $\mathinner{\!\left\lvert\lambda_{1}\right\rvert}\leq c/L^{2}$. We can restate this bound as follows. Denote

[TABLE]

Then we have

[TABLE]

provided that $\mathinner{\!\left\lvert\lambda_{1}\right\rvert}\leq c/L^{2}$.

Continuing in a similar way, we can show the following for every $k=1,\ldots,d$. Denote

[TABLE]

Then we have

[TABLE]

provided that $\mathinner{\!\left\lvert\lambda_{k-1}\right\rvert}\leq c/L^{2}$.

Step 3. Combining the increments using a martingale-like argument. Combining the pieces into a telescoping sum using Lemma 4.1, we obtain

[TABLE]

provided that

[TABLE]

If we choose $\lambda=\lambda_{0}\in\mathbb{R}$ so that $\mathinner{\!\left\lvert\lambda\right\rvert}\leq c_{0}/(dL^{2})$ with a sufficiently small absolute constant $c_{0}$, then we can show by induction that (5.3) holds and, moreover,

[TABLE]

We defer the verification of both of these bounds to Lemma 5.2 below. Substituting them into (5.2) and rearranging the terms, we conclude that

[TABLE]

Replacing $\lambda$ with $-\lambda$, we see that the same bound holds for $\operatorname{\mathbb{E}}\exp\left(-\lambda f^{2}+\lambda\operatorname{\mathbb{E}}f^{2}\right)$. Since the inequality $e^{\mathinner{\!\left\lvert z\right\rvert}}\leq e^{z}+e^{-z}$ holds for all $z\in\mathbb{R}$, we obtain

[TABLE]

Step 4. Deriving a concentration inequality for $f^{2}$. Using the exponential Markov’s inequality just like we did in the proof of Theorem 1.3 in Section 4, we get

[TABLE]

for any $u>0$ and any $0\leq\lambda\leq c_{0}/(dL^{2})$.

Let us optimize the right hand side in $\lambda$. A good choice is

[TABLE]

for a sufficiently small constant $c>0$. Indeed, if $c_{1}\leq c_{0}$ then $\lambda$ lies in the required range $0\leq\lambda\leq c_{0}/(dL^{2})$, and if $c_{1}\leq 1/(4C)$ then substituting this choice of $\lambda$ into our probability bound gives

[TABLE]

Step 5. Deriving a concentration inequality for $f$. Choose any $\varepsilon\geq 0$ and substitute $u\coloneqq\varepsilon\operatorname{\mathbb{E}}f^{2}$ into our probability bound. We get

[TABLE]

Now choose any $\delta\geq 0$ and apply this bound for $\varepsilon\coloneqq\max(\delta,\delta^{2})$. Then $\min(\varepsilon^{2},\varepsilon)=\delta^{2}$, and one can easily check the following implication

[TABLE]

(This follows from the implication $\mathinner{\!\left\lvert z-1\right\rvert}\geq\delta\Rightarrow\mathinner{\!\left\lvert z^{2}-1\right\rvert}\geq\max(\delta,\delta^{2})$ that is valid for all $z\geq 0.)$ Hence we obtain

[TABLE]

Now choose any $t\geq 0$ and apply this bound for $\delta\coloneqq t/(\operatorname{\mathbb{E}}f^{2})^{1/2}$. Recalling the value of $L$ from (5.1), we get

[TABLE]

Finally, we can use the theorem’s assumption on $t$ to get rid of the second exponential term just like we did in in the proof of Theorem 1.3 in Section 4. The proof is complete. ∎

In Step 3 of the argument above, we used the following bound on the multipliers $\lambda_{k}$, which we promised to prove later. Let us do it now.

Lemma 5.2** (Multipliers).**

Let $d,M\geq 0$ and consider a number $\lambda_{0}\in\mathbb{R}$ such that

[TABLE]

Define $\lambda_{1},\ldots,\lambda_{d}\in\mathbb{R}$ inductively by the formula

[TABLE]

Then, for every $k=1,\ldots,d$, we have:

[TABLE]

Proof.

We can prove the second inequality in the conclusion by induction. Assume that it holds for some $k$, i.e.

[TABLE]

By construction, the sequence $(\lambda_{k})$ is increasing, so the triangle inequality gives

[TABLE]

Furthermore, the assumption (5.4) implies that

[TABLE]

Substituting this into (5.6), we get

[TABLE]

Then we have

[TABLE]

Thus we proved (5.5) for $k+1$, so the second inequality in the conclusion is verified.

The first bound in the conclusion follows from the first. Indeed, using (5.7) and (5.6), we get

[TABLE]

as claimed. The proof is complete. ∎

6. Applications

In this section we state and prove a full version of Theorem 1.6 that states that random tensors are well conditioned. But before we do so, let us prove a result that may have an independent interest, namely a concentration inequality for the distance between a random tensor $X$ and a given subspace $L$.

Corollary 6.1** (Distance to a subspace).**

Let $n$ and $d$ be positive integers and $L\subset\mathbb{R}^{n^{d}}$ be a linear subspace with $k\coloneqq\operatorname{codim}(L)$. Consider a simple random tensor $X\coloneqq x_{1}\otimes\cdots\otimes x_{d}$ in $\mathbb{R}^{n^{d}}$, where all $x_{k}$ are independent random vectors in $\mathbb{R}^{n}$ whose coordinates are independent, mean zero, unit variance, subgaussian random variables. Then, for every $0\leq t\leq 2\sqrt{k}$, we have

[TABLE]

Here $c>0$ depends only on the bound on the subgaussian norms.

Proof.

Apply Theorem 1.4 for the orthogonal projection $P$ in $\mathbb{R}^{n^{d}}$ onto $L^{\perp}$ and note that $\mathinner{\!\left\lVert P\right\rVert}_{\mathsf{op}}=1$ and $\mathinner{\!\left\lVert P\right\rVert}_{\mathsf{HS}}=\sqrt{\dim(L^{\perp})}=\sqrt{k}$. ∎

For $d=1$, Corollary 6.1 recovers the known optimal concentration inequalities for the distance between a random vector and a fixed subspace are known (see e.g. [29, Corollary 2.1.19], [27], [35, Exercise 6.3.4]), which are frequently used in random matrix theory. Some previously known extensions for tensors of degrees $d\geq 2$ were given in [11, 9, 1, 8].

Now we are ready to state and prove a rigorous version of Theorem 1.6:

Corollary 6.2** (Random tensors are well conditioned).**

Consider independent simple subgaussian random tensors $X_{1},\ldots,X_{m}$ (defined like a tensor $X$ in Corollary 6.1). Let $\varepsilon$ be such that $Cd^{2}\log(n)/n\leq\varepsilon\leq 1/2$. If $m\leq(1-\varepsilon)n^{d}$ then, with probability at least $1-2\exp(-c\varepsilon n/d)$, we have

[TABLE]

Note that this result is nontrivial for $d=O(\sqrt{n/\log n})$, because only in this range is the range of $\varepsilon$ nonempty.

Proof.

We can assume that $\mathinner{\!\left\lVert a\right\rVert}_{2}=1$ without loss of generality. A simple “leave-one-out” bound gives

[TABLE]

where $L_{j}$ is the linear span of the $m-1$ vectors $(X_{i})_{i\neq j}$. Since $\dim(L_{j})\leq m-1\leq(1-\varepsilon)n^{d}$, we have $\operatorname{codim}(L_{j})\geq\varepsilon n^{d}$.

Fix $j$ and apply Corollary 6.1 with $t=\sqrt{\varepsilon n^{d}}/2$ conditionally on $(X_{i})_{i\neq j}$. It gives

[TABLE]

with probability at least $1-2\exp(-c\varepsilon n/2d)$. Taking the union bound over $j=1,\ldots,m$, we conclude that all events (6.2) hold smultaneously with probability at least

[TABLE]

where we used that $m\leq n^{d}$ and the assumption on $\varepsilon$. Substitute this into the leave-one-out bound (6.1) to complete the proof. ∎