A general theory of singular values with applications to signal denoising

TL;DR

This paper develops a general theory extending singular values to arbitrary finite-dimensional spaces with dual norms, with applications in various signal denoising techniques.

Contribution

It introduces a unified framework for singular values in general normed spaces, generalizing matrix and tensor decompositions for improved denoising methods.

Findings

Unified theory of singular values for dual norm spaces

Applications to diverse denoising problems like TV and LASSO

Enhanced understanding of signal decomposition and noise separation

Abstract

We study the Pareto frontier for two competing norms $\|\cdot\|_X$ and $\|\cdot\|_Y$ on a vector space. For a given vector $c$, the pareto frontier describes the possible values of $(\|a\|_X,\|b\|_Y)$ for a decomposition $c=a+b$. The singular value decomposition of a matrix is closely related to the Pareto frontier for the spectral and nuclear norm. We will develop a general theory that extends the notion of singular values of a matrix to arbitrary finite dimensional euclidean vector spaces equipped with dual norms. This also generalizes the diagonal singular value decompositions for tensors introduced by the author in previous work. We can apply the results to denoising, where $c$ is a noisy signal, $a$ is a sparse signal and $b$ is noise. Applications include 1D total variation denoising, 2D total variation Rudin-Osher-Fatemi image denoising, LASSO, basis pursuit denoising and tensor decompositions.

Equations

$$\Vert A{\Vert }_{\sigma }$$

$$\Vert A{\Vert }_{\ast }$$

$$\Vert A{\Vert }_2$$

$$e_1\otimes e_2\otimes e_3+e_1\otimes e_3\otimes e_2+e_2\otimes e_1\otimes e_3+e_2\otimes e_3\otimes e_1+e_3\otimes e_1\otimes e_2+e_3\otimes e_2\otimes e_1\in {\mathbb{R}}^{3\times 3\times 3}$$

$$e_2\otimes e_1\otimes e_1+e_1\otimes e_2\otimes e_1+e_1\otimes e_1\otimes e_2\in {\mathbb{R}}^{2\times 2\times 2}$$

$$f_{YX}^c:[0,\Vert c{\Vert }_X]\to [0,\Vert c{\Vert }_Y]$$

$$\Vert c{\Vert }_0=|\{i\mid 1\le i\le n,c_i\ne 0\}|=\underset{p\to 0}{\lim }\Vert c{\Vert }_p^p.$$

$$\Vert c-b{\Vert }_1={\sum }_{i=1}^n|c_i-b_i|\ge {\sum }_{i=1}^n\min \{|c_i|-y,0\}.$$

$$f_{XY}^c(y)=\Vert a{\Vert }_1={\sum }_{i=1}^n\max \{0,|c_i|-y\}.$$

$$\Vert v{\Vert }_Y=\max \{⟨v,w⟩\mid w\in V,\Vert w{\Vert }_X=1\}.$$

$$h_{YX}^c:[0,\Vert c{\Vert }_X]\to [0,\Vert c{\Vert }_Y].$$

$${\Vert \left(\begin{array}{c}{z}_1\\ z_2\end{array}\right)\Vert }_X=\sqrt{\frac{1}{2}{z}_1^2+2z_2^2}.$$

$${\Vert \left(\begin{array}{c}{z}_1\\ z_2\end{array}\right)\Vert }_Y=\sqrt{2z_1^2+\frac{1}{2}{z}_2^2}.$$

$$c=\left(\begin{array}{c}3\\ 3\end{array}\right)=a(t)+b(t)$$

$$a(t)=\left(\begin{array}{c}4-2t\\ 2t^{-1}-1\end{array}\right)\text{mbox}andb(t)=\left(\begin{array}{c}2t-1\\ 4-2t^{-1}\end{array}\right).$$

$$\Vert a(t){\Vert }_X=\sqrt{2}(2t^{-1}-1)\sqrt{t^2+1},\Vert b(t){\Vert }_Y=\sqrt{2}(2-t^{-1})\sqrt{t^2+1}$$

$$⟨a(t),b(t)⟩=(4-2t)(2t-1)+(2t^{-1}-1)(4-2t^{-1})=2(2t^{-1}-1)(2-t^{-1})(t^2+1)=\Vert a(t){\Vert }_X\Vert b(t){\Vert }_Y.$$

$$(\sqrt{2}(2t^{-1}-1)\sqrt{t^2+1},\sqrt{2}(2-t^{-1})\sqrt{t^2+1})),t\in [\frac{1}{2},2].$$

$$\tilde{a}(t)=\frac{1}{5}\left(\begin{array}{c}16-4t\\ 4t^{-1}-1\end{array}\right)\text{mbox}and\tilde{b}(t)=\frac{1}{5}\left(\begin{array}{c}4t-1\\ 16-4t^{-1}\end{array}\right).$$

$$(\frac{\sqrt{2}}{5}(4t^{-1}-1)\sqrt{4t^2+1},\frac{\sqrt{2}}{5}(4-t^{-1})\sqrt{t^2+4})).$$

$${\mu }_{XY}(c):=\frac{\Vert c{\Vert }_Y}{\Vert c{\Vert }_X}.$$

$$(x_1+\cdots +x_i,y_{i+1}+\cdots +y_r),i=0,1,2,\dots ,r.$$

$$f_{XY}^c(y)=\sum _{i=1}^n\max \{0,|c_i|-y\}$$

$$c=\left(\begin{array}{c}-1\\ 2\\ 4\\ 1\\ -2\\ 1\\ -1\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ 2\\ 0\\ 0\\ 0\\ 0\end{array}\right)+\left(\begin{array}{c}0\\ 1\\ 1\\ 0\\ -1\\ 0\\ 0\end{array}\right)+\left(\begin{array}{c}-1\\ 1\\ 1\\ 1\\ -1\\ 1\\ -1\end{array}\right)=c_1+c_2+c_3$$

$$(0,2+1+1)=(0,4),(2,1+1)=(2,2),(2+3,1)=(5,1),(2+3+7,0)=(12,0).$$

$$B_X(x)=\{v\in V\mid \Vert v{\Vert }_X\le x\}.$$

$${\mathrm{proj}}_X({\mathrm{proj}}_X(c,x_1),x_2)={\mathrm{proj}}_X(c,x_1)$$

$${\mathrm{shrink}}_X(c,x)=c-{\mathrm{proj}}_X(c,x).$$

$$a={\mathrm{proj}}_X(c,x)={\mathrm{shrink}}_Y(c,y)$$

$$b={\mathrm{proj}}_Y(c,y)={\mathrm{shrink}}_X(c,x).$$

$$c=c_1+c_2+\cdots +c_r$$

Full text

The author was partially supported by NSF grant DMS 1601229.

1. Introduction

Sound, images, and videos can be corrupted by noise. Noise removal is a fundamental problem in signal and image processing In the additive noise model, we have an original signal $a$, additive noise $b$ and a corrupted signal $c=a+b$. We will work with discrete signals and view $a$, $b$ and $c$ as vectors or arrays. The problem of noise removal can be framed in terms of competing norms on a vector space. The Pareto frontier defines the optimal trade-off between the two norms. The Pareto frontier was used in the L-curve method in Tikhonov regularization (see [44, 50, 27, 28] and [36, Chapter 26]), and in basis pursuit denoising (see [6, 48, 26]) to find optimal regularization parameters. The Pareto frontier is a continuous convex curve, and has a continuous derivative if one of the norms is the euclidean norm (see [6], Lemma 3.2 and Proposition 4.5).

We will assume that the original signal $a$ is sparse. A vector is sparse when it has few nonzero values. We will also consider other notions of sparseness. For example, a piecewise constant function on an interval can be considered sparse because its derivative has few nonzero values (or values where it is not defined), and a piecewise linear function can be considered sparse because the second derivative has few nonzero values. A sound signal from music is sparse because it contains only a few frequencies. A typical image is sparse, because it has large connected areas of the same color, i.e., the image is piecewise constant. This is exploited in the total variation image denoising method of Rudin, Osher and Fatemi ([46]). A matrix of low rank can also be viewed as sparse because it is the sum of a few rank 1 matrices. In this context, principal component analysis can be viewed as a method for recovering a sparse signal. The noise signal $b$, on the other hand, is not sparse. For example, white noise contains all frequences in the sound spectrum. gaussian additive noise in an image will be completely discontinuous and not locally constant at all.

There are many ways to measure sparseness. Examples of sparseness measures are the $\ell_{0}$ “norm” (which actually is not a norm), the rank of a matrix or the number of different frequencies in a sound signal. It is difficult to use these measure because they are not convex. We deal with this using convex relaxation, i.e., we replace the non-convex sparseness measure by a convex one. In this paper, we will measure sparseness using a norm $\|\cdot\|_{X}$ on the vector space of all signals. These norms coming from convex relaxation are typically $\ell_{1}$-type norms. For example, we may replace the $\ell_{0}$ “norm” by the $\ell_{1}$ norm, or replace the rank of a matrix by the nuclear norm. Noise will be measured using a different norm $\|\cdot\|_{Y}$. This typically will be a euclidean $\ell_{2}$ norm or perhaps an $\ell_{\infty}$ type norm. The quotient $\|a\|_{Y}/\|a\|_{X}$ is large for the sparse signal $a$, and $\|b\|_{Y}/\|b\|_{X}$ is small for the noise signal $b$. To denoise a signal $c$ we search for a decomposition $c=a+b$ where $\|a\|_{X}$ and $\|b\|_{Y}$ are small. Minimizing $\|a\|_{X}$ and $\|b\|_{Y}$ are two competing objectives. The trade-off between these two objectives is governed by the Pareto frontier. The concept of Pareto efficiency was used by Vilfredo Pareto (1848–1923) to decribe economic efficiency. A point $(x,y)\in{\mathbb{R}}^{2}$ is called Pareto efficient if there exists a decomposition $c=a+b$ with $\|a\|_{X}=x$ and $\|b\|_{Y}=y$ such that for every decomposition $c=a^{\prime}+b^{\prime}$ we have $\|a^{\prime}\|_{X}>x$, $\|b^{\prime}\|_{Y}>y$ or $(\|a^{\prime}\|_{X},\|b^{\prime}\|_{Y})=(x,y)$. If $(x,y)$ is Pareto efficient then we will call the decomposition $c=a+b$ an $XY$-decomposition. Many methods such as LASSO, basis pursuit denoising, the Dantzig selector, total variation denoising and principal component analysis can be formulated as finding an $XY$-decomposition for certain norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$.

In most examples that we consider, the space of signals has a positive definite inner product and the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are dual to each other. The inner product gives an euclidean norm defined by $\|c\|_{2}=\sqrt{\langle c,c\rangle}$. We now have $3$ different norms. We will show that $X2$-decompositions and $2Y$-decompositions are the same. We define the Pareto sub-frontier as the set of all points $(\|a\|_{X},\|b\|_{Y})$ where $c=a+b$ is an $X2$-decomposition (or equivalently, a $2Y$-decomposition). The Pareto sub-frontier lies on or above the Pareto frontier (by the definition of the Pareto frontier). A vector $c$ is called tight if its Pareto frontier and Pareto sub-frontier coincide. If every vector is tight then the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are called tight. We show that for tight vectors, the Pareto (sub-)frontier is piecewise linear. For a tight vector $c$, we will define the slope decomposition of $c$ which can be thought of as a generalization of the singular value decomposition.

The nuclear norm $\|\cdot\|_{\star}$ and spectral norm $\|\cdot\|_{\sigma}$ of a matrix are dual to each others (see for example [12, 45] and Lemma 7.1) and we will show that these norms are tight in Section 7. If the singular values of a matrix $A$ are $\lambda_{1},\lambda_{2},\dots,\lambda_{r}$ with multiplicities $m_{1},m_{2},\dots,m_{r}$ respectively, we have the following well-known formulas for the spectral, nuclear and euclidean (Frobenius) norm:

[TABLE]

The singular value region of $A$ is a bar of height $\lambda_{1}$ and width $m_{1}$, followed by a bar of height $\lambda_{2}$ and width $m_{2}$, etc. The singular values and their multiplicity can now easily be read off from the singular value region. For exampe if a matrix $A$ has eigenvalues $2.5$ with multiplicity 2, 1 with multiplicity 1 and 0.5 with multiplicity 2, then the singular value region is plotted below:

The height of the singular value region is the spectral norm, the width is the rank, the area is the nuclear norm, and if we integrate $2y$ over the region we obtain the square of the Frobenius norm $\|A\|_{2}^{2}$. The Pareto frontier of a matrix (which is also its Pareto sub-frontier) encodes the singular values of the matrix, and the slope decomposition is closely related to the singular value decomposition. The slope decomposition is unique, but the singular value decomposition may not be unique if some of the singular values coindice.

A tensor is a higher dimensional array. A $d$-way tensor is a vector for $d=1$ and a matrix for $d=2$. For $d\geq 3$, one can generalize the rank of a matrix to tensor rank ([31]). The tensor rank is closely related to the canonical polyadic decomposition of a tensor (CP-decomposition). This decomposition is also known as the PARAFAC ([29]) or the CANDECOMP model ([8]). The nuclear norm of a matrix can be generalized to the nuclear norm of a tensor ([25, 47]), and this can be viewed as a convex relaxation of the tensor rank. The spectral norm of a matrix can be generalized to a spectral norm of a tensor, and this norm is dual to the nuclear norm of a tensor ([19]). Not every tensor is tight. A tensor that is tight will have a slope decomposition which generalizes the diagonal singular value decomposition introduced by the author in [19]. Every tensor that has a diagonal singular value has a slope decomposition but the converse is not true. We will define singular values and multiplicities for tight tensors such that the formulas (1), (2), (3) are satisfied. The multiplicities of the singular values of tensors are nonnegative, but are not always integers. For example, in Section 13 we will show that the tensor

[TABLE]

is tight and has the singular value $\frac{2}{\sqrt{3}}$ with multiplicitiy $\frac{9}{2}$ (and not singular value $1$ with multiplicity 6 as one might expect). For tensors that are not tight we can still define the singular value region, but the singular value interpretation may me more esoteric. For example, we will show that the tensor

[TABLE]

has the following singular value region:

We will define the singular value region in a very general context. Whenever $V$ is a finite dimensional euclidean vector space, and $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are norms that are dual to each other, then we can define the singular value region for any $c\in V$.

Contents

1. 1 Introduction

2. 2 Main Results

3. 2.1 The Pareto frontier

4. 2.2 Dual norms and the Pareto sub-frontier

5. 2.3 The slope decomposition

6. 2.4 Geometry of the unit ball

7. 2.5 The Singular Value Region

8. 3 The Pareto Curve

9. 3.1 Optimization problems

10. 3.2 Properties of the Pareto curve

11. 3.3 Rigid norms

12. 3.4 The Pareto curve of sums

13. 4 Duality

14. 4.1 $X2$- and $2Y$-decompositions

15. 4.2 The Pareto sub-frontier

16. 4.3 Differentiating the Pareto curve

17. 5 Tight vectors

18. 5.1 An example of a norm that is not tight

19. 5.2 The Pareto sub-frontier of sums

20. 6 The slope decomposition

21. 6.1 unitangent vectors

22. 6.2 Faces of the unit ball

23. 6.3 Proof of Theorem 2.10

24. 6.4 Properties of the slope decomposition

25. 6.5 The unit ball of a tight norm

26. 7 The Singular Value Decomposition of a matrix

27. 7.1 Matrix norms

28. 7.2 Slope decomposition for matrices

29. 7.3 Principal Component Analysis

30. 7.4 The singular value region for matrices

31. 7.5 Rank minimization and low rank matrix completion

32. 8 Restricting Norms

33. 9 1D total variation denoising

34. 9.1 The total variation norm and its dual

35. 9.2 Description of the unit ball

36. 9.3 The taut string method

37. 9.4 An ECG example

38. 9.5 higher order total variation denoising

39. 10 The ISTA algorithm for $XY$-decompositions

40. 11 Basis Pursuit Denoising, LASSO and the Dantzig selector

41. 12 Total Variation Denoising in Imaging

42. 12.1 The 2D total variation norm

43. 12.2 Spareseness and total variation

44. 12.3 The total variation norm is not tight

45. 13 Tensor Decompositions

46. 13.1 CP decompositions

47. 13.2 The CoDe model and the nuclear norm

48. 13.3 Examples of unitangent tensors

49. 13.4 The diagonal SVD and the slope decomposition

50. 13.5 Group algebra tensors

51. 13.6 symmetric tensors in ${\mathbb{R}}^{2\times 2\times 2}$

2. Main Results

2.1. The Pareto frontier

Let us consider a finite dimensional ${\mathbb{R}}$-vector space $V$ equipped with two norms, $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$. Suppose that $c\in V$. We are looking for decompositions $c=a+b$ that are optimal in the sense that we cannot reduce $\|a\|_{X}$ without increasing $\|b\|_{Y}$ and we cannot reduct $\|b\|_{Y}$ without increasing $\|a\|_{X}$. We recall the definition from the introduction:

Definition 2.1**.**

A pair $(x,y)\in{\mathbb{R}}^{2}_{\geq 0}$ is called Pareto efficient if there exists a decomposition $c=a+b$ with $\|a\|_{X}=x$, $\|b\|_{Y}=y$ such that for every decomposition $c=a^{\prime}+b^{\prime}$ we have $\|a^{\prime}\|_{X}>x$, $\|b^{\prime}\|_{Y}>y$ or $(\|a^{\prime}\|_{X},\|b^{\prime}\|_{Y})=(x,y)$. If $(x,y)$ is a Pareto efficient pair then we call $c=a+b$ an $XY$-decomposition.

By symmetry, $c=a+b$ is an $XY$-decomposition if and only if $c=b+a$ is a $YX$-decomposition. The Pareto frontier consists of all Pareto efficient pairs (see [6]). The Pareto frontier is the graph of a strictly decreasing, continuous convex function

[TABLE]

(see [6] and Lemmas 3.2 and 3.3). If we change the role of $X$ and $Y$ we get the graph of $f_{XY}^{c}$, so $f_{XY}^{c}$ and $f_{YX}^{c}$ are inverse functions of each other.

Example 2.2**.**

Consider the vector space $V={\mathbb{R}}^{n}$. Sparseness of vectors in ${\mathbb{R}}^{n}$ can be measured by the number of nonzero entries. For $c\in V$ we define

[TABLE]

Note that $\|\cdot\|_{0}$ is not a norm on $V$ because it does not satisfy $\|\lambda c\|_{0}=|\lambda|\|c\|_{0}$ for $\lambda\in{\mathbb{R}}$. Convex relaxation of $\|\cdot\|_{0}$ gives us the $\ell_{1}$ norm $\|\cdot\|_{1}$. This means that the unit ball for the norm $\|\cdot\|_{1}$ is the convex hull of all vectors $c$ with $\|c\|_{0}=\|c\|_{2}=1$. Let us take $\|\cdot\|_{X}=\|\cdot\|_{1}$ and $\|\cdot\|_{Y}=\|\cdot\|_{\infty}$ and describe the Pareto frontier. Suppose that $c=(c_{1}\cdots c_{n})^{t}\in{\mathbb{R}}^{n}$ and $0\leq y\leq\|c\|_{\infty}$. If $\|b\|_{\infty}=y$ then we have

[TABLE]

If we take $b_{i}=\operatorname{sgn}(c_{i})\min\{|c_{i}|,y\}$ for all $i$, then we have equality. So $c=a+b$ is an $XY$-decomposition where $a:=c-b$. This shows that

[TABLE]

For the vector $c=(-1,2,4,1,-2,1,-1)^{t}$ we plotted $f_{XY}^{c}=f_{1\infty}^{c}$ and $f_{YX}^{c}=f_{\infty 1}^{c}$ (see also Example 2.11)

For example if we take $y=\frac{3}{2}$ then we get the decomposition with $a=(0,\frac{1}{2},\frac{5}{2},0,-\frac{3}{2},0,0)^{t}$ and $b=(-1,\frac{3}{2},\frac{3}{2},1,-\frac{3}{2},1,-1)^{t}$ and we have $\|a\|_{1}=\frac{7}{2}$ and $\|a\|_{0}=3$. The vector $a$ is sparser than $c$. This procedure of noise reduction is soft tresholding in its simplest form.

In Section 3, we will study the Pareto frontier and $XY$-decompositions in more detail.

2.2. Dual norms and the Pareto sub-frontier

We now assume that we have a positive definite bilinear form $\langle\cdot,\cdot\rangle$ on the finite dimensional vector space $V$. The euclidean $\ell_{2}$ norm on $V$ defined by $\|v\|_{2}:=\sqrt{\langle v,v\rangle}$. Suppose that $\|\cdot\|_{X}$ is another norm on $V$. We may think of $\|\cdot\|_{X}$ as a norm that measures sparseness. For denoising, we compare the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{2}$. We also consider the dual norm on $V$ defined by

[TABLE]

The dual norm of $\|\cdot\|_{Y}$ is $\|\cdot\|_{X}$ again. There is an interesting interplay between the 3 norms, and the $X2$-decompositions, $2Y$-decompositions and $XY$-decompositions are closely connected. The following proposition and other results in this section will be proved in Section 4.

Proposition 2.3**.**

For a vector $c\in V$, the following three statements are equivalent:

1. (1)

$c=a+b$* is an $X2$-decomposition;* 2. (2)

$c=a+b$* is a $2Y$-decomposition;* 3. (3)

$c=a+b$* and $\langle a,b\rangle=\|a\|_{X}\|b\|_{Y}$.*

Definition 2.4**.**

If the statements (1)–(3) in Proposition 2.3 imply

• (4)

$c=a+b$ is an $XY$-decomposition.

then $c$ is called tight. If all vectors in $V$ are tight then the norm $\|\cdot\|_{X}$ is called tight.

Definition 2.5**.**

The Pareto sub-frontier is the set of all pairs $(x,y)\in{\mathbb{R}}_{\geq 0}^{2}$ such that there exists an $X2$-decomposition $c=a+b$ with $\|a\|_{X}=x$ and $\|b\|_{Y}=y$.

We will show in Section 4 that the Pareto sub-frontier is the graph of a decreasing Lipschitz continuous function

[TABLE]

By symmetry, $h_{XY}^{c}$ is the inverse function of $h_{YX}^{c}$. From the definitions it is clear that $f_{YX}^{c}(x)\leq h_{YX}^{c}(x)$. If $f_{XY}^{c}=h_{XY}^{c}$, then every $X2$-decomposition is automatically a $XY$-decomposition, and $c$ is tight

Corollary 2.6**.**

A vector $c\in V$ is tight if and only if $f_{XY}^{c}=h_{XY}^{c}$.

If $f_{XY}^{c}=h_{XY}^{c}$ then there is no space between the two graphs and they fit together tightly, which explains the name of this property. Let us work out an example where the norms are not tight.

Example 2.7**.**

Define a norm $\|\cdot\|_{X}$ on ${\mathbb{R}}^{2}$ by

[TABLE]

Its dual norm is given by

[TABLE]

For $t\in[\frac{1}{2},2]$, the decomposition

[TABLE]

is an $X2$-decomposition, where

[TABLE]

To verify this, we compute

[TABLE]

and

[TABLE]

The Pareto sub-frontier is parameterized by

[TABLE]

The $XY$-decompositions of $c$ are $c=\widetilde{a}(t)+\widetilde{b}(t)$ for $t\in[\frac{1}{4},4]$, where

[TABLE]

The Pareto frontier is parameterized by

[TABLE]

We plotted the Pareto frontier in green and the Pareto sub-frontier in blue:

The Pareto frontier and sub-frontier are not the same, so the vector $c$ is not tight.

The Pareto sub-frontier encodes crucial information about the $X2$-decompositions. If $(x_{0},y_{0})$ is a point on the Pareto subfrontier and $c=a+b$ is an $X2$-decomposition with $\|a\|_{X}=x_{0}$ and $\|b\|_{Y}=y_{0}$, then we can read off $\|c\|_{X}$, $\|c\|_{Y}$, $\|c\|_{2}$, $\|a\|_{X}$, $\|a\|_{2}$, $\|b\|_{Y}$, $\|b\|_{2}$ and $\langle a,b\rangle$ from the Pareto sub-frontier using the following proposition.

Proposition 2.8**.**

Suppose that $(x_{0},y_{0})$ is a point on the $XY$-Pareto sub-frontier of $c\in V$, and let $c=a+b$ be an $X2$ decomposition with $\|a\|_{X}=x_{0}$ and $\|b\|_{Y}=y_{0}$.

1. (1)

The area below the sub-frontier is equal to $\frac{1}{2}\|c\|_{2}^{2}$. 2. (2)

The area below the sub-frontier and to the right of $x=x_{0}$ is equal to $\frac{1}{2}\|b\|_{2}^{2}$. 3. (3)

The area below the sub-frontier and above $y=y_{0}$ is equal to $\frac{1}{2}\|a\|_{2}^{2}$. 4. (4)

The area of the rectangle $[0,x_{0}]\times[0,y_{0}]$ is $\langle a,b\rangle$.

2.3. The slope decomposition

Proofs of results in this subsection will be given in Section 6. We define the slope of a nonzero vector $c\in V$ as the ratio

[TABLE]

If $\|\cdot\|_{X}$ is a norm that is small for sparse signals, then $\mu_{XY}$ is large for sparse signals, and small for noise. Note that $\mu_{YX}(c)=(\mu_{XY}(c))^{-1}$. Using the slope function, we define the slope decomposition. If $\|\cdot\|_{X}$ is the nuclear norm for matrices, then the slope decomposition is closely related to the singular value decomposition. So we can think of the slope decomposition as a generalization of the singular value decomposition.

Definition 2.9**.**

An expression $c=c_{1}+c_{2}+\cdots+c_{r}$ is called an $XY$-slope decomposition if $c_{1},\dots,c_{r}$ are nonzero, $\langle c_{i},c_{j}\rangle=\|c_{i}\|_{X}\|c_{j}\|_{Y}$ for all $i\leq j$ and $\mu_{XY}(c_{1})>\mu_{XY}(c_{2})>\cdots>\mu_{XY}(c_{r})$.

Note that because of symmetry, $c=c_{1}+c_{2}+\cdots+c_{r}$ is an $XY$ slope decomposition if and only if $c=c_{r}+c_{r-1}+\cdots+c_{1}$ is a $YX$ slope decomposition. There may be vectors that do not have a slope decomposition. We will prove the following result.

Theorem 2.10**.**

1. (1)

A vector $c\in V$ is tight if and only if $c$ has a slope decomposition. 2. (2)

Suppose that $c=c_{1}+c_{2}+\cdots+c_{r}$ is a slope decomposition, and let $x_{i}=\|c_{i}\|_{X}$ and $y_{i}=\|c_{i}\|_{Y}$ for all $i$. Then $\|c\|_{X}=\sum_{i=1}^{r}x_{i}$, $\|c\|_{Y}=\sum_{i=1}^{r}y_{i}$ and the Pareto frontier (which is the same as the sub-Pareto frontier) is the piecewise linear curve through the points

[TABLE]

Example 2.11**.**

Let us go back to Example 2.2. The norms $\|\cdot\|_{X}=\|\cdot\|_{1}$ and $\|\cdot\|_{Y}=\|\cdot\|_{\infty}$ are dual to each other. We will show that these norms are tight. If we integrate the function

[TABLE]

from [math] to $\|c\|_{Y}=\max\{|c_{1}|,|c_{2}|,\dots,|c_{n}|\}$, we get $\frac{1}{2}\sum_{i=1}^{n}c_{i}^{2}=\frac{1}{2}\|c\|_{2}^{2}$ which is the area under the graph of $h_{XY}^{c}$. So the areas under the graphs of $f_{XY}^{c}$ and $h_{XY}^{c}$ are the same, and we deduce that $h_{XY}^{c}=f_{XY}^{c}$. This shows that $c$ is tight. Since $c$ is arbitrary, the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are tight By Theorem 2.10 above, every vector has a slope decomposition. For example

[TABLE]

is a slope decomposition. Let $x_{i}=\|c_{i}\|_{1}$ and $y_{i}=\|c_{i}\|_{Y}$. Then we have $x_{1}=2$, $x_{2}=3$, $x_{3}=7$, $y_{1}=2$, $y_{2}=1$, $y_{3}=1$. The Pareto curve $f_{YX}^{c}$ is the piecewise linear function going through

[TABLE]

2.4. Geometry of the unit ball

For $x\geq 0$ we define the $X$-ball of radius $x$ by

[TABLE]

We explain denoising in terms of the geometry of the $X$-balls. Suppose we want to denoise a signal $c$, such that the denoised signal $a$ is sparse. We impose the constraint $\|a\|_{X}\leq x$. Under this constraint, we minimize the amount of noise, by minimizing the $\ell_{2}$ norm of $b:=c-a$. This means that the $a$ is the vector inside the ball $B_{X}(x)$ that is closest to the vector $c$. We call $a$ the projection of $c$ onto the ball $B_{X}(x)$ and write $a=\operatorname{proj}_{X}(c,x)$. The function $\operatorname{proj}_{X}(\cdot,x)$ is a retraction of ${\mathbb{R}}^{n}$ onto the ball $B_{X}(x)$. If $x_{1}\leq x_{2}$, then it is clear that

[TABLE]

For $x_{1}>x_{2}$ one might expect that $\operatorname{proj}_{X}(\operatorname{proj}_{X}(c,x_{1}),x_{2})=\operatorname{proj}_{X}(c,x_{2})$. This is not always true, but it is true in the case where $c$ is tight by Proposition 2.12 below.

We also define a shrinkage operator by

[TABLE]

If $c=a+b$ is an $X2$-decomposition, $\|a\|_{X}=x$ and $\|b\|_{Y}=y$ then we have

[TABLE]

and

[TABLE]

The function $\operatorname{shrink}_{Y}(\cdot,y)$ can be seen as a denoising function where $y$ is the noise level.

A nice property of tight vectors is the transitivity of denoising.

Proposition 2.12**.**

If $c$ is tight, then we have

1. (1)

$\operatorname{proj}_{X}(\operatorname{proj}_{X}(c,x_{1}),x_{2})=\operatorname{proj}_{X}(c,\min\{x_{1},x_{2}\})$* and* 2. (2)

$\operatorname{shrink}_{X}(\operatorname{shrink}_{X}(c,x_{1}),x_{2})=\operatorname{shrink}_{X}(c,x_{1}+x_{2})$.

The unit ball $B_{X}=B_{X}(1)$ is a closed convex set, but not always a polytope. We recall the definition of a face of a closed convex set:

Definition 2.13**.**

A face of a closed convex set $B\subseteq V$ is a convex closed subset $F\subseteq B$ with the following property: if $a,b\in B$, $0<t<1$ and $ta+(1-t)b\in F$ then we have $a,b\in F$.

We will study faces of the unit ball $B_{X}=B_{X}(1)$, and the cones associated to it.

Definition 2.14**.**

A facial $X$-cone is a cone of the form ${\mathbb{R}}_{\geq 0}F$ where $F\subset B_{X}$ is a (proper) face of the unit ball $B_{X}$. The set $\{0\}$ is considered a facial $X$-cone as well.

If $C$ is a nonzero facial $X$-cone, then $C={\mathbb{R}}_{\geq 0}F$ for some proper face $F$ of the unit ball $B_{X}$. In that case, we have $F=C\cap\partial B_{X}$, where $\partial B_{X}=\{c\in V\mid\|c\|_{X}=1\}$ is the unit sphere. We now will discuss two notions of sparseness related to a norm $\|\cdot\|_{X}$.

Definition 2.15**.**

For a nonzero vector $c$ we define its $X$-sparseness $\operatorname{sparse}_{X}(c)$ as the smallest nonnegative integer $r$ such that we can write

[TABLE]

where $c_{i}/\|c_{i}\|_{X}$ is a extremal point of the unit ball $B_{X}$ for $i=1,2,\dots,r$. The geometric $X$-sparseness $\operatorname{gsparse}_{X}(c)$ is $\dim C$ where $C$ is the smallest facial $X$-cone containing $c$.

Each notion of sparseness has its merits. We have

[TABLE]

but a similar inequality does not always hold for geometric sparseness. On the other hand, the set

[TABLE]

of $k$-geometric $X$-sparse vectors is closed, but the set

[TABLE]

of $k$-$X$-sparse vectors is not always closed.

We have

[TABLE]

by the Carathéodory Theorem (see [4, Theorem 2.3]).

Example 2.16**.**

Consider ${\mathbb{R}}^{n}$, and let $\|\cdot\|_{X}=\|\cdot\|_{1}$ and $\|\cdot\|_{Y}=\|\cdot\|_{\infty}$. We have

[TABLE]

where

[TABLE]

The function $\|\cdot\|_{0}$ is sometimes referred to as the $\ell_{0}$ norm, but is strictly speaking not a norm.

[TABLE]

We have

[TABLE]

but

[TABLE]

Example 2.17**.**

If $\|\cdot\|_{X}=\|\cdot\|_{\star}$ is the nuclear norm on a space $V={\mathbb{R}}^{m\times n}$ of $m\times n$ matrices. and $c\in V$ is a matrix, then $\operatorname{sparse}_{X}(c)=\operatorname{gsparse}_{X}(c)=\operatorname{rank}(c)$ is the rank of $c$.

Example 2.18**.**

Let $V={\mathbb{R}}^{2\times 2\times 2}={\mathbb{R}}^{2}\otimes{\mathbb{R}}^{2}\otimes{\mathbb{R}}^{2}$ be the tensor product space. The value $\operatorname{sparse}_{X}(c)$ is the tensor rank of $c\in V$. It is known that the set of all tensors of rank $\leq 2$ is not closed. If $\{e_{1},e_{2}\}$ is the standard orthogonal basis of ${\mathbb{R}}^{2}$, then the tensor

[TABLE]

has rank $3$, but is a limit of tensors of rank $2$. In this case, the geometric sparseness and sparseness are not the same. For example, if

[TABLE]

then $\operatorname{sparse}_{X}(d)=2$ is the tensor rank, but $\operatorname{gsparse}_{X}(d)>2$.

Theorem 2.19**.**

Suppose that $c=a+b$ is an $X2$-decomposition. Then we have

[TABLE]

where $n=\dim V$. Moreover, if $c$ is tight then we have

[TABLE]

since $\operatorname{gsparse}_{X}\leq\operatorname{sparse}_{X}$, the theorem also implies that

[TABLE]

2.5. The Singular Value Region

We can generalize the notion of the singular values region to an arbitrary finite dimensional vector space $V$ with dual norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$. Let $y=h_{YX}^{c}(x)$ be the Pareto sub-frontier of $c\in V$. For the definition of the singular value region, we view $x$ as a function of $y$ which gives $x=h_{XY}^{c}(y)$. The function $h_{XY}^{c}(y)$ is Lipschitz and decreasing and is differentiable almost everywhere.

Definition 2.20**.**

The singular value region is the region in the first quadrant to the left of the graph of $x=-\frac{d}{d\,y}h_{XY}^{c}(y)$.

If the graph of $-\frac{d}{d\,y}h_{XY}^{c}(y)$ is a step function, then we can interpret the region as singular values with multiplicities similarly as in the case of matrices. If a vector $c\in V$ has a slope decomposition, then we can easily find the singular values and multiplicities from Theorem 2.10. Recall that $\mu_{XY}(c)=\|c\|_{Y}/\|c\|_{X}$ and $\mu_{YX}(c)=\|c\|_{X}/\|c\|_{Y}$.

Corollary 2.21**.**

If

[TABLE]

is an $XY$ slope decomposition, then the singular values are

[TABLE]

with multiplicities

[TABLE]

respectively.

As we will see in Section 7, this notion of the singular value region is indeed a generalization of the singular value region defined for matrices in the introduction.

3. The Pareto Curve

3.1. Optimization problems

We will formulate signal denoising as an optimization problem. Suppose that $V$ is an $n$-dimensional ${\mathbb{R}}$-vector space equipped with two norms, $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$. Let $c\in V$ be a fixed vector. Given a noisy signal $c$ we would like to find a decomposition $c=a+b$ where both $\|a\|_{X}$ and $\|b\|_{Y}$ is small. We can do this by minimizing $\|a\|_{X}$ under the constraint $\|b\|_{Y}\leq y$ for some fixed $y\geq 0$, or by minimizing $\|b\|_{Y}$ under the constraint $\|a\|_{X}\leq x$ for some $x$ with $0\leq x\leq\|c\|_{X}$, we formally define the following optimization problem:

Problem ${\bf M}_{YX}^{c}(x)$: *Minimize $\|c-a\|_{Y}$ for $a\in V$ under the constraint $\|a\|_{X}\leq x$.

This has a solution because the function $a\mapsto\|c-a\|_{Y}$ is continuous, and the domain $\{a\in{\mathbb{R}}^{n}\mid\|a\|_{X}\leq x\}$ is compact. Let $f_{YX}^{c}(x)$ be the smallest value of $\|c-a\|_{Y}$. As we will see later in Lemma 3.3, the graph of $f_{YX}^{c}(x)$ consists of all Pareto efficient pairs $(x,y)$, and we call $f_{YX}^{c}$ the Pareto curve or Pareto frontier. Since $c$ will be fixed most of the time, we may just write ${\bf M}_{YX}(x)$ and $f_{YX}(x)$ instead of ${\bf M}_{YX}^{c}(x)$ and $f_{YX}^{c}(x)$. We will prove some properties of the Pareto curve.

Lemma 3.1**.**

If $0\leq x\leq\|c\|_{X}$ and $a$ is a solution to ${\bf M}_{YX}^{c}(x)$, then $\|a\|_{X}=x$.

Proof.

Suppose that $a$ is a solution to ${\bf M}_{YX}(x)$ and $\|a\|_{X}<x$. Note that $\|a\|_{X}<x\leq\|c\|_{X}$, so $c\neq a$ and $\|c-a\|_{Y}>0$. Choose $\varepsilon>0$ such that $\|a\|_{X}+\varepsilon\|c\|_{X}\leq x$ and define $a^{\prime}=(1-\varepsilon)a+\varepsilon c$. Then we have $\|a^{\prime}\|_{X}\leq(1-\varepsilon)\|a\|_{X}+\varepsilon\|c\|_{X}\leq x$ and

[TABLE]

Contradiction. ∎

3.2. Properties of the Pareto curve

The following lemma gives some basic properties of the Pareto curve. In various contexts, these properties are already known. We formulate the properties for the problem of two arbitrary competing norms on a finite dimensional vector space.

Lemma 3.2**.**

The function $f_{YX}$ is a convex, strictly decreasing, continuous function on the interval $[0,\|c\|_{X}]$.

Proof.

Suppose that $0\leq x_{1}<x_{2}\leq\|c\|_{X}$. There exist $a_{1},a_{2}\in V$ with $\|a_{i}\|_{X}=x_{i}$ and $\|c-a_{i}\|_{Y}=f_{YX}(x_{i})$ for $i=1,2$.

Let $a=(1-t)a_{1}+ta_{2}$ for some $t\in[0,1]$. Then we have $\|a\|_{X}=\|(1-t)a_{1}+ta_{2}\|_{X}\leq(1-t)x_{1}+tx_{2}$, so by definition we have

[TABLE]

This proves that $f$ is convex.

For some $t\in(0,1]$ we can write $x_{2}=(1-t)x_{1}+t\|c\|_{X}$. We have

[TABLE]

This shows that $f$ is strictly decreasing.

Since $f_{YX}$ is convex, it is continuous on $(0,\|c\|_{X})$. Because $f$ is decreasing, it is continuous at $x=\|c\|_{X}$. We will show that $f$ is also continuous at $x=0$. Suppose that $x_{1},x_{2},\dots$ is a sequence in $[0,\|c\|_{X}]$ with $\lim_{n\to\infty}x_{n}=0$. Choose $a_{n}\in V$ such that $\|a_{n}\|_{X}=x_{n}$ and $\|c-a_{n}\|_{Y}=f_{YX}(x_{n})$. Since $\lim_{n\to\infty}\|a_{n}\|_{X}=\lim_{n\to\infty}x_{n}=0$, we have that $\lim_{n\to\infty}a_{n}=0$. It follows that $\lim_{n\to\infty}f_{YX}(x_{n})=\lim_{n\to\infty}\|c-a_{n}\|_{Y}=\|c\|_{Y}=f_{YX}(0)$ because $\|\cdot\|_{Y}$ is a continuous function. ∎

We show now that the graph of $f_{YX}$ consists of all Pareto efficient pairs.

Lemma 3.3**.**

Suppose that $c\in V$.

1. (1)

$c=a+b$* is an $XY$-decomposition if and only if $a$ is a solution to ${\bf M}_{YX}^{c}(\|a\|_{X})$.* 2. (2)

The pair $(x,y)$ is Pareto efficient if and only if $0\leq x\leq\|c\|_{X}$ and $y=f^{c}_{YX}(x)$.

Proof.

(1) Suppose that $c=a+b$. If $c=a+b$ is an $XY$-decomposition, then it is clear from the definitions that $a$ is a solution to ${\bf M}_{YX}^{c}(\|a\|_{X})$.

On the other hand, suppose that $a$ is a solution to ${\bf M}_{YX}^{c}(\|a\|_{X})$ and $c=a^{\prime}+b^{\prime}$. Let $x=\|a\|_{X}$ and $y=\|b\|_{Y}$. Assume that $\|a^{\prime}\|_{X}\leq\|a\|_{X}=x$. Then we have $\|b^{\prime}\|_{Y}=\|c-a^{\prime}\|_{Y}\geq\|c-a\|_{Y}=\|b\|_{Y}=y$. If $\|b^{\prime}\|_{Y}=y$, then $a^{\prime}$ is also a solution to ${\bf M}_{YX}^{c}(x)$ and $\|a^{\prime}\|_{X}=x$ by Lemma 3.1. This shows that $c=a+b$ is an $XY$-decomposition.

(2) Suppose that $(x,y)$ is Pareto efficient. Assume that $\|c\|_{X}<x$. From the decomposition $c=c+0$ follows that $0=\|0\|_{Y}>y$. Contradiction. This shows that $0\leq x\leq\|c\|_{X}$. There exists a decomposition $c=a+b$ with $\|a\|_{X}=x$ and $\|c-a\|_{Y}=\|b\|_{Y}=y$. Because $\|a\|_{X}\leq x$, we have $y=\|c-a\|_{Y}\geq f^{c}_{YX}(x)$. Suppose that $a^{\prime}$ is a solution of ${\bf M}^{c}_{XY}(x)$. Then $\|a^{\prime}\|_{X}=x$ by Lemma 3.1. Because $(x,y)$ is Pareto efficient, we have $f^{c}_{YX}(x)=\|c-a^{\prime}\|_{Y}\geq y$. We conclude that $f_{YX}^{c}(x)=y$.

Conversely, suppose that $0\leq x\leq\|c\|_{X}$ and $y=f^{c}_{YX}(x)$. Let $a$ be a solution of ${\bf M}_{YX}^{c}(x)$ and $b:=c-a$. Suppose that $c=a^{\prime}+b^{\prime}$ is another decomposition. If $\|a^{\prime}\|_{X}<x$, then we have $\|b^{\prime}\|_{Y}=\|c-a^{\prime}\|_{Y}\geq f^{c}_{YX}(\|a^{\prime}\|_{X})>f^{c}_{YX}(x)=y$. If $\|a^{\prime}\|_{X}=x$ then we have $\|b^{\prime}\|_{Y}=\|c-a^{\prime}\|_{Y}\geq f^{c}_{YX}(\|a^{\prime}\|_{X})=f^{c}_{YX}(x)=y$. We conclude that $(x,y)$ is Pareto efficient. ∎

Corollary 3.4**.**

1. (1)

The function $f_{YX}:[0,\|c\|_{X}]\to[0,\|c\|_{Y}]$ is a homeomorphism and its inverse is $f_{XY}$. 2. (2)

A vector $a$ is a solution to ${\bf M}_{YX}(x)$ if and only if $c-a$ is a solution to ${\bf M}_{XY}(f_{YX}(x))$.

Proof.

(1) If $0\leq x\leq\|c\|_{X}$ and $0\leq y\leq\|c\|_{Y}$ then we have

[TABLE]

So$f_{XY}$ and $f_{YX}$ are inverse of each other. Since both functions are continuous, the functions are homeomorphisms.

If $y=f_{XY}(x)$ and $b=c-a$ then we have

[TABLE]

∎

3.3. Rigid norms

The problem ${\bf M}_{YX}^{c}(x)$ does not always have a unique solution.

Definition 3.5**.**

We say that $c\in V$ is rigid if ${\bf M}_{YX}^{c}(x)$ has a unique solution for all $x\in[0,\|c\|_{X}]$.

Let us give an example of a vector that is not rigid.

Example 3.6**.**

Suppose $V={\mathbb{R}}^{2}$ and $\|c\|_{X}=\|c\|_{Y}=\|c\|_{\infty}$ for all $c\in V$. Then $(1,1)^{t}$ is rigid because for $x\in[0,1]$ the vector $(x,x)^{t}$ is the unique solution to ${\bf M}^{c}_{YX}(x)$. The vector $c=(1,0)^{t}$ is not rigid: If $0<x<1$ then ${\bf M}^{c}_{YX}(x)$ has infinitely many solutions, namely $(x,s)^{t}$, $|s|\leq\min\{x,1-x\}$.

If $c\in V$ is rigid, then we can study how the unique solution of ${\bf M}_{YX}^{c}(x)$ varies as we change the value of $x$. The lemma below shows that the solution varies continuously. In various contexts, this property is well-known and used in homotopy continuation methods (see for example [42, 43, 21, 6]) for some optimization problem.

Lemma 3.7**.**

Suppose that $c\in V$ is rigid and let $\alpha_{YX}(x)=\alpha_{YX}^{c}(x)$ be the unique solution to ${\bf M}_{YX}^{c}(x)$ for $x\in[0,\|c\|_{X}]$. Then $\alpha_{YX}:[0,\|c\|_{X}]\to V$ is continuous.

Proof.

Suppose that $x_{1},x_{2},\dots\in[0,\|c\|_{X}]$ is a sequence for which $x=\lim_{n\to\infty}x_{n}$ exists. We assume that $\lim_{n\to\infty}\alpha_{YX}(x_{n})$ does not exist, or that it is not equal to $\alpha_{YX}(x)$. By replacing $x_{1},x_{2},\dots$ by a subsequence, we may assume that $a=\lim_{n\to\infty}\alpha_{YX}(x_{n})$ exists, but that it is not equal to $\alpha_{YX}(x)$. We have $\|a\|_{X}=\lim_{n\to\infty}\|\alpha_{YX}(x_{n})\|_{X}=\lim_{n\to\infty}x_{n}=x$. Also, we get $\|c-a\|_{Y}=\lim_{n\to\infty}\|c-\alpha_{YX}(x_{n})\|_{Y}=\lim_{n\to\infty}f_{YX}(x_{n})=f_{YX}(x)$ because $f$ is continuous. Because ${\bf M}_{YX}(x)$ has a unique solution, we conclude that $a=\alpha_{YX}(x)$. Contradiction. We conclude that $\lim_{n\to\infty}\alpha_{YX}(x_{n})=\alpha_{YX}(x)$. This proves that $\alpha_{YX}$ is continuous. ∎

Definition 3.8**.**

The norm $\|\cdot\|_{Y}$ is called strictly convex if $\|a+b\|_{Y}=\|a\|_{Y}+\|b\|_{Y}$ implies that $a$ and $b$ are linearly dependent.

The $\ell_{2}$-norm on ${\mathbb{R}}^{n}$ is strictly convex, for example.

Lemma 3.9**.**

If $\|\cdot\|_{Y}$ is strictly convex, then every vector is rigid.

Proof.

Suppose that $\|\cdot\|_{Y}$ is strictly convex and that $c\in V$. We will prove that $c$ is rigid. Suppose $\|a\|_{X}=\|a^{\prime}\|_{X}=x$ and $\|b\|_{Y}=\|b^{\prime}\|_{Y}=f_{YX}(x)$, where $b=c-a$ and $b^{\prime}=c-a^{\prime}$. Let $\overline{a}=(a+a^{\prime})/2$. Then we have $\|\overline{a}\|_{X}\leq\frac{1}{2}(\|a\|_{X}+\|a^{\prime}\|_{X})=x$. By definition, we have $\|c-\overline{a}\|_{Y}\geq f_{YX}(x)$. It follows that

[TABLE]

Since $\|\cdot\|_{Y}$ is strictly convex, $b$ and $b^{\prime}$ are linearly dependent. Because $\|b\|_{Y}=\|b^{\prime}\|_{Y}$, it follows that $b=\pm b^{\prime}$. If $b=-b^{\prime}$ then we have $b+b^{\prime}=0$. From $\|b\|_{Y}+\|b^{\prime}\|_{Y}\leq\|b+b^{\prime}\|_{Y}=0$ follows that $b=b^{\prime}=0$ and $a=a^{\prime}=c$ and the uniqueness is established. If $b=b^{\prime}$, then $a=a^{\prime}$ and we have again uniqueness. ∎

3.4. The Pareto curve of sums

Next we will compare the Pareto curve of $f_{YX}^{a+b}$ with the Pareto curves $f_{YX}^{a}$ and $f_{YX}^{b}$. For this purpose, we introduce the concatenation of two functions. Suppose that $u:[0,t]\to{\mathbb{R}}$ and $v:[0,s]\to{\mathbb{R}}$ are two functions with $u(t)=v(s)=0$. The concatenation $u\star v:[0,s+t]\to{\mathbb{R}}$ is defined by

[TABLE]

Note that $(u\star v)(0)=u(0)+v(0)$ and $(u\star v)(s+t)=v(s)=0$. If $u$ and $v$ are decreasing, then so is $u\star v$. If $u$ and $v$ are continuous, then so is $u\star v$. Note that concatenation is associative: $(u\star v)\star w=u\star(v\star w)$. However, it is not commutative.

Example 3.10**.**

Suppose that $u,v:[0,1]\to{\mathbb{R}}$ are defined by $u(x)=1-x$ and $v=(1-x)^{2}$:

The graphs of $u\star v$ and $v\star u$ are:

Lemma 3.11**.**

Suppose that $a,b\in V$. If $0\leq x\leq\|a+b\|_{X}$ then we have

[TABLE]

Proof.

Suppose that $0\leq x\leq\|a\|_{X}$. Choose $d\in V$ such that $\|d\|_{X}=x$ and $\|a-d\|_{Y}=f_{YX}^{a}(x)$. Then we have

[TABLE]

Reversing the roles of $X$ and $Y$, and $a$ and $b$ gives

[TABLE]

if $0\leq y\leq\|b\|_{Y}$. Substituting $y=f_{YX}^{a+b}(x)$ where $\|a\|_{X}\leq x\leq\|a+b\|_{X}$ yields

[TABLE]

Applying the decreasing function $f_{YX}^{b}$ gives

[TABLE]

∎

4. Duality

4.1. $X2$- and $2Y$-decompositions

Suppose that the vector space $V$ is equipped with a positive definite bilinear form $\langle\cdot,\cdot\rangle$ and a norm $\|\cdot\|_{X}$. The bilinear form gives an $\ell_{2}$-norm $\|\cdot\|_{2}$ and let $\|\cdot\|_{Y}$ be the dual norm of $\|\cdot\|_{X}$. In this section we will study $X2$-decompositions, which turn out to be the same as $2Y$-decompositions. We start with the following characterization of an $X2$-decomposition:

Proposition 4.1**.**

The expression $c=a+b$ is an $X2$-decomposition if and only if $\langle a,b\rangle=\|a\|_{X}\|b\|_{Y}$.

Proof.

Suppose that $c=a+b$ is an $X2$-decomposition. Choose a vector $d$ such that $\langle b,d\rangle=\|b\|_{Y}$ and $\|d\|_{X}=1$. Let $\varepsilon>0$. Define $a^{\prime}=(1-\varepsilon)a+\varepsilon\|a\|_{X}d$ and $b^{\prime}=c-a^{\prime}$. We have $\|a^{\prime}\|_{X}\leq(1-\varepsilon)\|a\|_{X}+\varepsilon\|a\|_{X}=\|a\|_{X}$. Therefore, $\|b^{\prime}\|_{2}\geq\|b\|_{2}$. It follows that

[TABLE]

Taking the limit $\varepsilon\downarrow 0$ yields the inequality

[TABLE]

so $\langle a,b\rangle\geq\|a\|_{X}\|b\|_{Y}$. The opposite inequality $\langle a,b\rangle\leq\|a\|_{X}\|b\|_{Y}$ holds because the norms are dual to each other. We conclude that $\langle a,b\rangle=\|a\|_{X}\|b\|_{Y}$.

Conversely, suppose that $\langle a,b\rangle=\|a\|_{X}\|b\|_{Y}$. Let $x=\|a\|_{X}$, let $a^{\prime}$ be the solution to ${\bf M}_{2X}^{c}(x)$ and define $b^{\prime}=c-a^{\prime}$. Then $c=a^{\prime}+b^{\prime}$ is an $X2$-decomposition.

[TABLE]

So we conclude that $a=a^{\prime}$, and $c=a+b$ is an $X2$-decomposition. ∎

The equivalence between the $X2$-decomposition and $2Y$-decomposition (Proposition 2.3) now easily follows.

Proof of Proposition 2.3.

In Proposition 2.3, (1) and (3) are equivalent because of Proposition 4.1. Dually, (2) and (3) are equivalent. ∎

4.2. The Pareto sub-frontier

We define $h_{YX}=h^{c}_{YX}:[0,\|c\|_{X}]\to[0,\|c\|_{Y}]$ by $h_{YX}(x)=\|c-\alpha_{2X}(x)\|_{Y}$. The graph of $h_{YX}^{c}$ is the Pareto subfrontier. Indeed, if $c=a+b$ is an $X2$-decomposition with $\|a\|_{X}=x$ and $\|b\|_{Y}=y$, then we have $\alpha_{2X}(x)=a$ and $h_{YX}(x)=\|c-\alpha_{2X}(x)\|_{Y}=\|c-a\|_{Y}=\|b\|_{Y}=y$. We now prove some properties of the Pareto sub-frontier.

Lemma 4.2**.**

We have $\alpha_{2Y}(h_{YX}(x))=c-\alpha_{2X}(x)$.

Proof.

Let $a=\alpha_{2X}(x)$ and $b=c-a$. Then $c=a+b$ is an $X2$-decomposition, therefore also a $2Y$-decomposition. So $c=b+a$ is a $Y2$-decomposition. Let $y=\|b\|_{Y}=h_{YX}(x)$. Then we have $b=\alpha_{2Y}(y)=\alpha_{2Y}(h_{YX}(x))$. ∎

Lemma 4.3**.**

The function $h_{YX}(x)$ is a strictly decreasing homeomorphism and its inverse is $h_{XY}(x)$.

Proof.

We have have

[TABLE]

The function $\alpha_{2X}$ is injective, because the function $f_{2X}(x)=\|c-\alpha_{2X}(x)\|_{2}$ is injective. It follows that $h_{XY}(h_{YX}(x))=x$. By symmetry, we also have $h_{YX}(h_{XY}(y))=y$, so $h_{XY}$ is the inverse of $h_{YX}$.

The function $h_{YX}(x)=\|c-\alpha_{2X}(x)\|_{Y}$ is continuous, because $\alpha_{2X}$ and $\|\cdot\|_{Y}$ are continuous. This proves that $h_{YX}$ is a homeomorphism. By the Intermediate Value Theorem, it has to be strictly increasing or strictly decreasing. Since $h_{YX}(\|c\|_{X})=0\leq h_{YX}(0)=\|c\|_{Y}$, the function $h_{YX}$ must be strictly decreasing. ∎

Proposition 4.4**.**

The function $h_{YX}$ is Lipschitz continuous.

Proof.

Since $\|\cdot\|_{2}$ and $\|\cdot\|_{X}$ are norms on a finite dimensional vector space, there exists positive constant $K>0$ such that $\|c\|_{Y}\leq K\|c\|_{2}$. Suppose that $c=a_{1}+b_{1}$ and $c=a_{2}+b_{2}$ are $XY$-decompositions, with $x_{2}:=\|a_{2}\|_{X}>x_{1}:=\|a_{1}\|_{X}$. It follows that $y_{2}:=\|b_{2}\|_{Y}<y_{1}:=\|b_{1}\|_{Y}$. We have

[TABLE]

We conclude that

[TABLE]

∎

4.3. Differentiating the Pareto curve

The function $f_{2X}(x)$ is differentiable. A special case (but with a similar prove) was treated in [6, §2].

Proposition 4.5**.**

The function $f_{2X}(x)$ is differentiable on $[0,\|c\|_{X}]$, and

[TABLE]

Proof.

If $0\leq x_{1},x_{2}\leq\|c\|_{X}$ then we have

[TABLE]

Reversing the roles of $x_{1},x_{2}$ gives us

[TABLE]

if $x_{2}>x_{1}$ then we obtain

[TABLE]

and if $x_{1}>x_{2}$ then we have

[TABLE]

Since $h_{YX}$ is continuous, it follows that $(f_{2X})^{2}$ is differentiable on $[0,\|c\|_{X}]$ with derivative $-2h_{YX}$. Since $f_{2X}(x)$ is positive on $[0,\|c\|_{X})$, it is differentiable on $[0,\|c\|_{X})$. We have

[TABLE]

∎

Proof of Proposition 2.8.

From Proposition 4.5 follows that

[TABLE]

So the area to the right of $x=\|a\|_{X}$ is

[TABLE]

Similarly, the area below the graph and above the line $y=\|b\|_{Y}$ is equal to $\|a\|_{2}^{2}$. The area below the graph of $h_{YX}$ is equal to $\frac{1}{2}\|c\|_{2}^{2}=\frac{1}{2}\|a\|_{2}^{2}+\frac{1}{2}\|b\|_{2}^{2}+\langle a,b\rangle$. The area of the $\|a\|_{X}\times\|b\|_{Y}$ rectangle is $\|a\|_{X}\|b\|_{Y}=\langle a,b\rangle$. ∎

The solution for ${\bf M}^{c}_{2X}(\lambda)$ can be obtained from a regularized quadratic minimization problem.

Proposition 4.6**.**

The vector $a$ is a solution to ${\bf M}_{2X}^{c}(\lambda)$ if and only if

[TABLE]

is minimal.

Proof.

We can choose $a$ such that $\|c-a\|_{Y}+\frac{1}{2\lambda}\|a\|_{2}^{2}$ is minimal. Let $b=c-a$. Then $c=a+b$ is an $X2$-decomposition, so $\langle a,b\rangle=\|a\|_{X}\|b\|_{Y}$. The function

[TABLE]

has a minimum at $t=1$. So we have

[TABLE]

and $\|a\|_{X}=\lambda$. This shows that $a$ is solution ${\bf M}_{2X}^{c}(\lambda)$. Since $M_{2X}^{c}(\lambda)$ has a unique solution, this unique solution $a$ must minimize $\|c-a\|_{Y}+\frac{1}{2\lambda}\|a\|_{2}^{2}$. ∎

A similar argument shows that $a$ is a solution to ${\bf M}_{2X}^{c}(\lambda)$ if and only if $\|c-a\|_{Y}+\frac{1}{\lambda}\|a\|_{2}$ is minimal.

5. Tight vectors

Suppose that $V$ is an $n$-dimensional vector space with a positive definite bilinear form, and that $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are norms which are dual to each other. From the definitions it is clear that $f_{YX}^{c}(x)\leq h_{YX}^{c}(x)$. Recall that $c$ is tight if we have equality for all $x\in[0,\|c\|_{X}]$. If $c\in V$ is tight and rigid, then $\alpha_{2X}(x)=\alpha_{YX}(x)$ for $x\in[0,\|c\|_{X}]$. If every vector in $V$ is tight, then $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are called tight norms. In this section we study properties of tight vectors and tight norms.

5.1. An example of a norm that is not tight

Consider the norm on ${\mathbb{R}}^{2}$ defined by $\|c\|_{X}=\max\{|c_{2}|,|2c_{2}-c_{1}|\}$ for $c=(c_{1},c_{2})^{t}$. Its dual norm is given by $\|c\|_{Y}=\max\{|c_{1}+c_{2}|,|3c_{1}+c_{2}|\}$. The unit balls are polar duals of each other:

Consider the vector $c=(3,12)^{t}$. Below are the functions $f_{YX}^{c}$ and $h_{YX}^{c}$. We see that $f^{c}_{YX}$ and $h^{c}_{YX}$ are not the same, so $c$ is not tight. The example shows that $h_{YX}^{c}$ is not always convex.

The trajectories of $a_{YX}^{c}$ (green) and $a_{2X}^{c}$ (blue) are sketched in the graph below.

For every positive value of $x$, $a_{2X}^{c}(x)/x$ lies on the unit ball $B_{X}$. In fact, $a_{2X}^{c}(x)/x$ is the vector in $B_{X}$ that is closest to $c/x$ with respect to the euclidean distance. In the graph below, $c/x$ and $a_{2X}^{c}(x)/x$ are plotted for for various values of $x$. Note that $a_{2X}^{c}(x)/x$ is constant on the intervals $(0,1]$ and $[3,6]$. On these intervals $a_{2X}^{c}(x)$ moves on a line through the origin. On the other intervals $[1,3]$ and $[6,21]$, $a_{2X}^{c}(x)$ moves on a line through $c$.

The singular value region for the vector $c$ is as follows:

If we want to iterpret the singular value region in terms of singular values, we must allow negative multiplicities. The singular values of $c$ are 21 with multiplicity $0.1$, $11$ with multiplicity $0.9$, $9$ with multiplicity $-0.5$ and $3$ with multiplicity $4.5$.

5.2. The Pareto sub-frontier of sums

We have defined the concatenation of two graphs, and now we define the concatenation of two paths in $V$ in a similar manner. If $\beta:[0,t]\to V$ and $\gamma:[0,s]\to V$ are curves with $\beta(0)=\gamma(0)=0$, then we define the concatenation $\beta\star\gamma:[0,s+t]\to V$ by

[TABLE]

Theorem 5.1**.**

Suppose that $c\in V$ is tight and that $c=a+b$ is an $XY$-decomposition.

1. (1)

$a$* and $b$ are tight as well;* 2. (2)

$\|c\|_{X}=\|a\|_{X}+\|b\|_{X}$* and $\|c\|_{Y}=\|a\|_{Y}+\|b\|_{Y}$;* 3. (3)

$f_{YX}^{c}=f_{YX}^{a}\star f_{YX}^{b}$; 4. (4)

$\alpha_{YX}^{c}=\alpha_{YX}^{a}\star\alpha_{YX}^{b}$.

Proof.

Suppose that $0\leq x\leq\|a\|_{X}$. Choose a vector $d$ such that $\|d\|_{X}=x$ and $\|a-d\|_{Y}=f_{YX}^{a}(x)$. We have

[TABLE]

Suppose that one of the inequalities above is strict for some $x\in[0,\|a\|_{X}]$. Integrating $t$ from [math] to $x$ yields

[TABLE]

Contradiction. We conclude that $h_{YX}^{a}(x)=f_{YX}^{a}(x)=h_{YX}^{c}(x)-\|b\|_{Y}$ for all $x\in[0,\|a\|_{X}]$. In particular, $a$ is tight. By symmetry, $b$ is tight as well. This proves (1). For $x=0$ we get $\|a\|_{Y}=h_{YX}^{a}(0)=h_{YX}^{c}(0)-\|b\|_{Y}=\|c\|_{Y}-\|b\|_{Y}$. So we have $\|c\|_{Y}=\|a\|_{Y}+\|b\|_{Y}$ and by symmetry we also have $\|c\|_{X}=\|a\|_{X}+\|b\|_{X}$. This proves (2).

If $0\leq x\leq\|a\|_{X}$, then we have

[TABLE]

By symmetry, if $0\leq y\leq\|b\|_{Y}$, then we have

[TABLE]

It follows that

[TABLE]

Substituting $y=f_{YX}^{c}(x)$ yields

[TABLE]

This proves (3).

If $x\in[0,\|a\|_{X}]$, then we have

[TABLE]

So we get

[TABLE]

If $x\in[\|a\|_{X},\|c\|_{X}]$ then we have

[TABLE]

and

[TABLE]

So for all $x\in[0,\|c\|_{X}]$ we have

[TABLE]

and

[TABLE]

So $(\alpha_{YX}^{a}\star\alpha_{YX}^{b})(x)$ is the unique solution to ${\bf M}_{YX}^{c}(x)$ and therefore equal to $\alpha_{YX}^{c}(x)$. This proves (4). ∎

Proof of Proposition 2.12.

We already now part (1) in the case $x_{1}\leq x_{2}$. Assume that $x_{1}>x_{2}$. Let $c=a+b$ be the $XY$-decomposition with $\|a\|_{X}=x_{1}$. This is also a $X2$-decomposition and $a=\operatorname{proj}_{X}(c,x_{1})$. Now $a$ and $b$ are tight by Theorem 5.1(1). Let $a=a_{1}+a_{2}$ be an $XY$-decomposition with $\|a_{1}\|_{X}=x_{2}$. Now $a=a_{1}+a_{2}$ is also an $X2$-decomposition and $a_{1}=\operatorname{proj}_{X}(a,x_{2})=\operatorname{proj}_{X}(\operatorname{proj}_{X}(c,x_{1}),x_{2})$. We have $a_{1}=\alpha_{YX}^{a}(x_{1})=(\alpha_{YX}^{a}\star\alpha_{YX}^{b})(x_{1})=\alpha_{YX}^{c}(x_{1})$. So $c=a_{1}+(c-a_{1})$ is an $XY$-decomposition (and $X2$-decomposition) and $a_{1}=\operatorname{proj}_{X}(c,x_{2})$. This proves that $\operatorname{proj}_{X}(c,x_{2})=\operatorname{proj}_{X}(\operatorname{proj}_{X}(c,x_{1}),x_{2})$ and part (1) has been proved.

Suppose that $c=a+b$ is an $X2$-decomposition with $\|a\|_{X}=x_{1}$. Then $b=\operatorname{shrink}_{X}(c,x_{1})$. Let $b=b_{1}+b_{2}$ be an $X2$-decomposition with $\|b_{1}\|_{X}=x_{2}$. Then $b_{2}=\operatorname{shrink}_{X}(b,x_{2})=\operatorname{shrink}_{X}(\operatorname{shrink}_{X}(c,x_{1}),x_{2})$. Similar reasoning as before shows that $c=(a+b_{1})+b_{2}$ is an $X2$-decomposition. Also $(a+b_{1})$ is tight and $(a+b_{1})=a+b_{1}$ is an $XY$-decomposition, so $\|a+b_{1}\|_{X}=\|a\|_{X}+\|b_{1}\|_{X}=x_{1}+x_{2}$. Therefore, $b_{2}=\operatorname{shrink}_{X}(c,x_{1}+x_{2})$ and we are done. ∎

Lemma 5.2**.**

If $c=a+b$ is an $X2$-decomposition and $a$ and $b$ are tight, then we have $h^{c}_{YX}=h^{a}_{YX}\star h^{b}_{YX}$.

Proof.

Suppose that $0\leq x\leq\|a\|_{X}$ and let $d=\alpha_{YX}^{a}(x)=\alpha_{2X}^{a}(x)$. We get

[TABLE]

It follows that

[TABLE]

So $c=d+(c-d)$ is an $X2$-decomposition and $\|c-d\|_{Y}=\|a-d\|_{Y}+\|b\|_{Y}$. We get

[TABLE]

Suppose that $\|a\|_{X}\leq x\leq\|c\|_{X}$ and define $y=h^{c}_{YX}(x)$. Then we have $0\leq y\leq\|b\|_{Y}$ and by symmetry we get

[TABLE]

and

[TABLE]

We conclude that $h^{c}_{YX}=h^{a}_{YX}\star h_{YX}^{b}$. ∎

We will show later in Proposition 6.11 that under the assumptions of Lemma 5.2 the vector $c$ is tight.

6. The slope decomposition

6.1. unitangent vectors

In this section we study the slope decomposition. We show that a vector $c$ is tight if and only if it has a slope decomposition. We also will show that the Pareto frontier is always piecewise linear for a tight vector. In that case, the different slopes in the Pareto frontier correspond to different summands in the slope decomposition of $c$.

For a vector $c\in V$ we have the inequality $\|c\|_{2}^{2}=\langle c,c\rangle\leq\|c\|_{X}\|c\|_{Y}$.

Definition 6.1**.**

We call a vector $c\in V$ unitangent if $\|c\|_{2}^{2}=\|c\|_{X}\|c\|_{Y}$.

Unitangent vectors are the simplest kind of tight vectors. As we will see, their Pareto frontier is linear, i.e., has only one slope. Recall that $\|c\|_{Y}$ is the maximal value of the functional $\langle c,\cdot\rangle$ on the unit ball $B_{X}$. Now $c$ is unitangent if and only if the maximum of $\langle c,\cdot\rangle$ is attained at $c/\|c\|_{X}\in B_{X}$.

Proposition 6.2**.**

If $c\in V$ is unitangent, then it is tight and we have $\alpha_{YX}^{c}(x)=(x/\|c\|_{X})c$ and $f_{YX}^{c}(x)=\|c\|_{Y}(1-x/\|c\|_{X})$ for $x\in[0,\|c\|_{X}]$.

Proof.

Suppose that $\|a\|_{X}\leq x$. Then we have

[TABLE]

It follows that

[TABLE]

Since $a$ was arbitrary, we conclude that $f_{YX}^{c}(x)\geq\|c\|_{Y}(1-x/\|c\|_{X})$.

If we take $a=(x/\|c\|_{X})c$, then we have $\|a\|_{X}=x$ and $\|c-a\|_{Y}=\|c\|_{Y}(1-x/\|c\|_{X})\leq f_{YX}^{c}(x)$. We conclude that $f_{YX}^{c}(x)=\|c\|_{Y}(1-x/\|c\|_{X})$ and $\alpha_{YX}^{c}(x)=a=(x/\|c\|_{X})c$. ∎

6.2. Faces of the unit ball

Suppose that $B$ is a compact convex subset of a finite dimensional ${\mathbb{R}}$-vector space $V$. Recall that a convex closed subset $F$ of $B$ is a face if $v,w\in B$, $0<t<1$ and $tv+(1-t)w\in F$ implies that $v,w\in F$. The following lemma is easily shown by induction and is left to the reader.

Lemma 6.3**.**

If $F$ is a face of $B$, $v_{1},v_{2},\dots,v_{r}\in B$, $t_{1},t_{2},\dots,t_{r}>0$ such that $t_{1}+t_{2}+\cdot+t_{r}=1$ and $t_{1}v_{1}+\cdots+t_{r}v_{r}\in F$, then $v_{1},v_{2},\dots,v_{r}\in F$.

Proof.

We prove the statement by induction on $r$. This is clear from the definition of a face for $r\leq 2$. Suppose $r>2$. Let $t_{i}^{\prime}=t_{i}/(1-t_{r})$ and $w=t_{1}^{\prime}v_{1}+\cdots+t_{r-1}^{\prime}v_{r-1}$. We have $t_{r}v_{r}+(1-t_{r})w=t_{1}v_{1}+\cdots+t_{r}v_{r}\in F$, so $v_{r},w\in F$. Since $t_{1}^{\prime}+\cdots+t_{r-1}^{\prime}=1$ and $w=t_{1}^{\prime}v_{1}+\cdots+t_{r-1}^{\prime}v_{r-1}\in F$ we get $v_{1},\dots,v_{r-1}$ by induction. ∎

Lemma 6.4**.**

If $B\subseteq V$ is a compact convex set, then the smallest face containing $v\in B$ is

[TABLE]

Proof.

Suppose that $w_{1},w_{2}\in F$ and $sw_{1}+(1-s)w_{2}\in B$ for some $s>0$. There exist $t>0$ such that $(1+t)v-tw_{1},(1+t)v-tw_{2}\in B$, so we have

[TABLE]

and therefore $sw_{1}+(1-s)w_{2}\in F$. This proves that $F$ is a face of $B$.

Suppose that $F^{\prime}$ is any face of $B$ containing $v$. If $w\in F$ then there exists $t>0$ such that $(1+t)v-tw\in B$. Since $v\in F^{\prime}$ is a convex combination of $(1+t)v-tw,w\in B$, we have $w,(1+t)v-tw\in F^{\prime}$. So $F^{\prime}$ contains $F$. ∎

Let $B_{X}$ be the unit ball for the norm $\|\cdot\|_{X}$.

Lemma 6.5**.**

A convex cone $C$ in $V$ is a facial $X$-cone if and only if it has the following norm-sum property: If $a,b\in V$, $a+b\in C$ and $\|a+b\|_{X}=\|a\|_{X}+\|b\|_{X}$ then we have $a,b\in C$.

Proof.

Suppose that $C$ is a cone in $V$. If $C=\{0\}$ then $C$ is a facial $X$-cone and has the norm-sum property. Assume now that $C\neq\{0\}$. Let $\partial B_{X}=\{c\in V\mid\|c\|_{X}=1\}$ be the unit sphere. Take $F=C\cap\partial B_{X}$ so that $C={\mathbb{R}}_{\geq 0}F$. We have to show that $F$ is a face of $B_{X}$ if and only if $C$ has the norm-sum property.

Suppose that $F$ is a face. If $a+b\in C$ and $\|a+b\|_{X}=\|a\|_{X}+\|b\|_{X}$ then we have

[TABLE]

where $t=\|a\|_{X}/\|a+b\|_{X}$ and $a/\|a\|_{X},b/\|b\|_{X},(a+b)/\|a+b\|_{X}\in F$. If $t=0$ or $t=1$ then $a=0$ or $b=0$ and $a,b\in C$. Otherwise, $0<t<1$ and $a/\|a\|_{X},b/\|b\|_{X}\in F$ because $F$ is a face. We conclude that $a,b\in C$. So $C$ has the norm-sum property.

Conversely, suppose that $C$ has the norm-sum property, $a,b\in B_{X}$ and $0<t<1$ such that $ta+(1-t)b\in F$ then we have

[TABLE]

and the inequalities are equalities. It follows that $\|a\|_{X}=\|b\|_{X}=1$ The norm-sum property gives $ta,(1-t)b\in C$, so $a,b\in C$. We conclude that $a,b\in C\cap\partial B_{X}=F$. ∎

Lemma 6.6**.**

Suppose that $a\in V$ is nonzero, and $C$ is the set of all $b\in V$ for which there exists $\varepsilon>0$ such that $\|a-\varepsilon b\|_{X}+\|\varepsilon b\|_{X}=\|a\|_{X}$. Then $C$ is the smallest facial $X$-cone containing $a$.

Proof.

If $a\in V$, then we have $\|a-\varepsilon b\|_{X}+\|\varepsilon b\|_{X}=\|a\|_{X}$ and every facial $X$ cone containing $a$ must also contain $\varepsilon b$ and $b$ by Lemma 6.5. Now $C$ itself is a facial $X$ cone: if $b_{1},b_{2}\in C$ then there exists $\varepsilon_{1},\varepsilon_{2}>0$ such that $\|a-\varepsilon_{i}b_{i}\|_{X}+\|\varepsilon_{i}b_{i}\|_{X}=\|a\|_{X}$ for $i=1,2$. We can replace $\varepsilon_{1}$ and $\varepsilon_{2}$ by the minimum of the two and assume that $\varepsilon_{1}=\varepsilon_{2}=\varepsilon$. We have

[TABLE]

We must have equalities everywhere, so $\frac{1}{2}(b_{1}+b_{2})$ and $b_{1}+b_{2}$ lie in $C$ by Lemma 6.5. ∎

Definition 6.7**.**

For $c\in V$ and $0<x<\|c\|_{X}$ we define ${\mathcal{F}}_{X}(x)$ as the smallest face of $B_{X}$ containing $\alpha_{2X}(x)/x$.

Lemma 6.8**.**

Suppose that $c$ is a tight vector. If $0<x_{1}<x_{2}<\|c\|_{X}$ then we have ${\mathcal{F}}_{X}(x_{1})\subseteq{\mathcal{F}}_{X}(x_{2})$.

Proof.

Suppose that $c$ is tight and $0<x_{1}<x_{2}<\|c\|_{X}$. For $t=x_{2}/(x_{2}-x_{1})$ we have

[TABLE]

which lies in the unit ball $B_{X}$. This proves that $\alpha_{YX}(x_{1})/x_{1}$ lies in ${\mathcal{F}}_{X}(x_{2})$. We conclude that ${\mathcal{F}}_{X}(x_{1})\subseteq{\mathcal{F}}_{X}(x_{2})$. ∎

Proposition 6.9**.**

If $c\in V$ is tight, then $\alpha_{YX}(x)$ and $f_{YX}(x)$ are piecewise linear.

Proof.

We can divide up the interval $[0,\|c\|_{X}]$ into finitely many intervals such that on each interval ${\mathcal{F}}_{X}$ is constant. Suppose that $(x_{1},x_{2})$ is an open interval on which ${\mathcal{F}}_{X}$ equal to $F$. The affine hull of $F$ is of the form $d+W$ where $W\subset V$ is a subspace and $d\in W^{\perp}$. Now $\alpha_{YX}(x)$ is the vector in $xd+W$ closest to $c$ (in the euclidean norm). If we define $a(x)=\alpha_{YX}(x)-dx$, then $a(x)\in W$ is the vector closest to $c-dx$. So $a(x)$ is the orthogonal projection of $c-dx$ onto $W$. Since $d\in W^{\perp}$, $a(x)$ is the orthogonal projection of $c$ onto $W$, so $a(x)=a$ is constant. This proves that $\alpha_{YX}(x)=a+dx$ is linear.

Because $\langle c-a,a\rangle=0$, we have

[TABLE]

So $h_{YX}(x)$ is linear for $x\in(x_{1},x_{2})$. ∎

Proof of Theorem 2.19.

Suppose that $c=a+b$ is an $X2$-decomposition and let $x=\|a\|_{X}$ and $y=\|b\|_{Y}$. The smallest face of $B_{X}$ containing $x^{-1}a$ is ${\mathcal{F}}_{X}(x)$ and the smallest face containing $y^{-1}b$ is ${\mathcal{F}}_{Y}(y)$. For every $a^{\prime}\in{\mathcal{F}}_{X}(x)$ and every $b^{\prime}\in{\mathcal{F}}_{Y}(y)$ we have $\langle a^{\prime},b^{\prime}\rangle\leq 1$. We have $\langle x^{-1}a,y^{-1}b\rangle=1$. Since $y^{-1}b$ lies in the relative interior of ${\mathcal{F}}_{Y}(y)$, we have $\langle x^{-1}a,b^{\prime}\rangle=1$ for all $b^{\prime}\in{\mathcal{F}}_{Y}(y)$. Since $x^{-1}a$ lies in the relative interior of ${\mathcal{F}}_{X}(x)$, we have $\langle a^{\prime},b^{\prime}\rangle=1$ for all $a^{\prime}\in{\mathcal{F}}_{X}(x)$ and all $b^{\prime}\in{\mathcal{F}}_{Y}(y)$. It follows that

[TABLE]

If $c$ is tight, them we have ${\mathcal{F}}_{X}(x)\subseteq{\mathcal{F}}_{X}(\|c\|_{X})$ and

[TABLE]

∎

6.3. Proof of Theorem 2.10

Suppose that $c$ is tight. Then $h^{c}_{YX}=f^{c}_{YX}$ is piecewise linear by Proposition 6.9. Suppose that $z_{0}<z_{1}<\cdots<z_{r}=\|c\|_{X}$ such that $h^{c}_{YX}$ is linear on each interval $[z_{i-1},z_{i}]$, and that $h^{c}_{YX}$ is not differentiable at $z_{1},\dots,z_{r-1}$. Let $a_{i}=\alpha_{2X}^{c}(z_{i})$, and define $c_{i}=a_{i}-a_{i-1}$ for $i=1,2,\dots,r$. We have $a_{i-1}=\alpha_{2X}^{c}(z_{i-1})=\alpha_{2X}^{a_{i}}(z_{i-1})$ so $a_{i}=a_{i-1}+c_{i}$ is an $X2$-decomposition for all $i$. By induction we get

[TABLE]

The area under the graph of $f^{c_{i}}_{YX}=h^{c_{i}}_{YX}$ is ${\textstyle\frac{1}{2}}\|c_{i}\|_{2}^{2}$. The area under the graph of $f^{c_{1}}_{YX}\star\cdots\star f^{c_{r}}_{YX}$ is

[TABLE]

So we have

[TABLE]

This proves that $\langle c_{i},c_{j}\rangle=\|c_{i}\|_{X}\|c_{j}\|_{Y}$ for all $i\leq j$. This shows that $c=c_{1}+c_{2}+\cdots+c_{r}$ is a slope decomposition.

Conversely, suppose that $c=c_{1}+c_{2}+\cdots+c_{r}$ is a slope decomposition. We will show that $c$ is tight. Since $c_{1}+\cdots+c_{r-1}$ is also a slope decomposition, so by induction we may assume that $c_{1}+\cdots+c_{r-1}$ is tight, and

[TABLE]

Since $c=(c_{1}+\cdots+c_{r-1})+c_{r}$ is an $X2$-decomposition, it follows from Lemma 5.2 that

[TABLE]

Suppose that

[TABLE]

We have

[TABLE]

So we have

[TABLE]

It follows that

[TABLE]

We get $f^{c}_{YX}=h^{c}_{YX}$, so $c$ is tight. We have proven part (1).

(2) Suppose that $c$ is tight and $c=c_{1}+c_{2}+\cdots+c_{r}$ is a slope decomposition. Then the graph $f^{c_{i}}_{YX}$ is a straight line segment from $(0,y_{i})$ to $(x_{i},0)$ where $y_{i}=\|c_{i}\|_{Y}$ and $x_{i}=\|c_{i}\|_{X}$. Since $f^{c}_{YX}=f^{c_{1}}_{YX}\star f^{c_{2}}_{YX}\star\cdots\star f^{c_{r}}_{YX}$, we have that $f^{c}_{YX}$ is the graph through the points $(x_{1}+\cdots+x_{i},y_{i+1}+\cdots+y_{r})$ for $i=0,1,2,\dots,r$.

6.4. Properties of the slope decomposition

Lemma 6.10**.**

If $c=c_{1}+\cdots+c_{m}$ is an $XY$-slope decomposition, then $c_{1},\dots,c_{m}$ are linearly independent.

Proof.

Suppose that we have

[TABLE]

Because $\mu_{XY}(c_{i})=\|c_{i}\|_{Y}/\|c_{i}\|_{X}>\mu_{XY}(c_{m})=\|c_{m}\|_{Y}/\|c_{m}\|_{X}$ we get

[TABLE]

Contradiction. This proves that $c_{1},\dots,c_{r}$ are linearly independent. ∎

Proposition 6.11**.**

Suppose that $c=a+b$ is an $X2$-decomposition and $a$ and $b$ are tight. Then $c$ is tight.

Proof.

Since $a$ and $b$ are tight, they have slope decompositions, say $a=a_{1}+\cdots+a_{r}$ and $b=b_{1}+\cdots+b_{s}$. We have

[TABLE]

It follows that $\langle a_{i},b_{j}\rangle=\|a_{i}\|_{X}\|b_{j}\|_{Y}$ for all $i<j$. If $\mu_{XY}(a_{r})>\mu_{XY}(b_{1})$ then

[TABLE]

is a slope decomposition.

Suppose that $\mu_{XY}(a_{r})\leq\mu_{XY}(b_{1})$. We have

[TABLE]

so $\mu_{XY}(a_{r})\geq\mu_{XY}(a_{r}+b_{1})$. Similarly, we have

[TABLE]

so $\mu_{XY}(a_{r}+b_{1})\geq\mu_{XY}(b_{1})\geq\mu_{XY}(a_{r})\geq\mu_{XY}(a_{r}+b_{1})$. We conclude that $\mu_{XY}(a_{r})=\mu_{XY}(b_{1})=\mu_{XY}(a_{r}+b_{1})$ and

[TABLE]

is a slope decomposition.

Since $c$ has a slope decomposition, it is tight. ∎

6.5. The unit ball of a tight norm

Proposition 6.12**.**

A norm $\|\cdot\|_{X}$ is tight if and only if every face $F$ of the unit ball $B_{X}$ contains a unitangent vector that is perpendicular to $F$.

Proof.

Suppose that $\|\cdot\|_{X}$ is tight and that $F$ is a face of $B_{X}$ (other than $B_{X}$ itself). Choose $c\in F$ in the relative interior. Then we have a slope decomposition

[TABLE]

If $t_{i}=\|c_{i}\|_{X}/\|c\|_{X}$ then we have $t_{1}+t_{2}+\cdots+t_{r}=1$ and

[TABLE]

Now $c_{1},\dots,c_{r}\in F$ by Lemma 6.3. We have $\langle c_{r},c\rangle=\|c\|_{X}\|c_{r}\|_{Y}=\|c_{r}\|_{Y}$. For any other vector $b\in B_{X}$ we have $\langle c_{r},b\rangle\leq\|b\|_{X}\|c_{r}\|_{Y}\leq\|c_{r}\|_{Y}$. So the functional $\langle c_{r},\cdot\rangle$ on F is maximal at $c$, and therefore maximal and constant on the face $F$. It follows that $c_{r}$ is perpendicular to $F$.

Now we show the converse. Suppose that every face $F$ of the unit ball $B_{X}$ contains a unitangent vector that is perpendicular to $F$. Suppose that $c\in V$ is a vector with $\|c\|_{X}=1$. Let $F$ be the smallest face of $B_{X}$ that contains $c/\|c\|_{X}$. By induction on $\dim F$ we show that $c$ is tight. The case $\dim F=0$ is clear. Suppose that $\dim F>0$. There exists a vector $b\in F$ that is unitangent and orthogonal to $c$. Choose $t$ maximal such that $v=c+t(c-b)=(1+t)c-tb\in F$. Let $a^{\prime}=\frac{1}{1+t}v$ and $b^{\prime}=\frac{t}{t+1}b$. Then we have $c=a^{\prime}+b^{\prime}$. Since $v$ lies in a face of smaller dimension, we know by induction that $v$ and $a^{\prime}$ are tight. Because $b^{\prime}$ is unitangent, it is also tight. We have

[TABLE]

[TABLE]

[TABLE]

So $\langle a^{\prime},b^{\prime}\rangle=\|a^{\prime}\|_{X}\|b^{\prime}\|_{Y}$. It follows that $c=a^{\prime}+b^{\prime}$ is an $X2$-decomposition. By Lemma 6.10, $c$ is tight. ∎

Example 6.13**.**

Consider again Examples 2.11 and 2.2. Suppose that $c=(c_{1},\dots,c_{n})^{t}\in{\mathbb{R}}^{n}$ and define $\lambda_{1}>\lambda_{2}>\cdots>\lambda_{r}>0$ by

[TABLE]

Define $m_{i}$ be the multiplicity of $\lambda_{i}$, i.e., $m_{i}$ is the number of values of $j$ for which $|c_{j}|=\lambda_{i}$. We define vectors $c^{(1)},\dots,c^{(r)}\in{\mathbb{R}}^{n}$ as follows:

[TABLE]

and

[TABLE]

We use the convention $\lambda_{r+1}=0$. We have

[TABLE]

We have $\|c^{(i)}\|_{\infty}=\lambda_{i}-\lambda_{i+1}$. and

[TABLE]

If $i<j$ then we have

[TABLE]

We have

[TABLE]

This shows that (4) is a slope decomposition. So the norms $\|\cdot\|_{\infty}$ and $\|\cdot\|_{1}$ are tight.

7. The Singular Value Decomposition of a matrix

7.1. Matrix norms

In this section, we will study the singular value decomposition of a matrix using our terminology and the results we have obtained. Let $V={\mathbb{C}}^{m\times n}$ be the space of $m\times n$-matrices. We have a bilinear form $\langle\cdot,\cdot\rangle$ on $V$ defined by

[TABLE]

where $B^{\star}$ denotes the conjugate transpose of $B$ and $\Re(\cdot)$ denotes the real part. We will study 3 norms on the vector space $V$ namely the euclidean norm, the nuclear norm and the spectral norm, and express each of these in terms of the singular values of a matrix.

The matrix $A^{\star}A$ is a nonnegative definite Hermitian matrix and its eigenvalues are nonnegative and real. The euclidean $\ell_{2}$-norm of a matrix is given by

[TABLE]

Since $A^{\star}A$ is positive semi-definite Hermitian, we can choose a unitary matrix $U$ such that $U^{\star}A^{\star}AU$ is a diagonal matrix with diagonal entries $\lambda_{1}^{2},\lambda_{2}^{2},\dots,\lambda_{n}^{2}$ where $\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{n}\geq 0$ are the singular values of $A$. We have

[TABLE]

Let $D$ be the diagonal matrix

[TABLE]

Then $UDU^{\star}$ is the unique positive semi-definite Hermitian matrix whose square is $A^{\star}A$, and we will denote this matrix by $\sqrt{A^{\star}A}$.

We define the spectral norm or operator norm $\|A\|_{\sigma}$ of $A$ by

[TABLE]

We have

[TABLE]

The nuclear norm $\|\cdot\|_{\star}$ is defined by

[TABLE]

The proof of the following well-known result will be useful for the discussion that follows.

Lemma 7.1**.**

The norms $\|\cdot\|_{\sigma}$ and $\|\cdot\|_{*}$ are dual to each other.

Proof.

Let $A\in V$. We use the notation as before, where $\sqrt{A^{\star}A}=UDU^{\star}$, and $D$ is the diagonal matrix whose diagonal entries are the singular values $\lambda_{1},\lambda_{2},\dots,\lambda_{n}$. Let $u_{1},u_{2},\dots,u_{n}$ be the columns of $U$. These vectors form an orthonormal basis and for any $B\in V$ we have

[TABLE]

Because $(AU)^{\star}(AU)=D^{2}$, the columns of $AU$ are orthogonal. The matrices $W=AUD^{-1}$ is unitary. So $A=WDU^{\star}$. If $w_{1},\dots,w_{n}$ are the columns of $W$, then the singular value decomposition of $A$ is

[TABLE]

Let $r$ be maximal such that $\lambda_{r}\neq 0$ and define the block matrix

[TABLE]

For $B=WEU^{\star}$ we have

[TABLE]

From (5) and (6) follows that $\|\cdot\|_{\star}$ is the dual norm of $\|\cdot\|_{\sigma}$. ∎

7.2. Slope decomposition for matrices

Suppose that $C$ is a complex $m\times n$ matrix and let $\lambda_{1}>\lambda_{2}>\dots>\lambda_{r}>0$ be the nonsingular values of $C$ with multiplicities $m_{1},m_{2},\dots,m_{r}$ respectively. We can write $C=WDU^{\star}$ where $U,W$ are unitary and $D$ is of the form

[TABLE]

where $I_{d}$ is the $d\times d$ identity matrix and the zeros represent possible empty zero blocks. The norms can be expressed as follows:

[TABLE]

Define

[TABLE]

for $j=1,2,\dots,r$ where $k_{j}=m_{1}+m_{2}+\cdots+m_{j}$ and $\lambda_{r+1}=0$. We have

[TABLE]

Proposition 7.2**.**

The expression (7) is a slope decomposition. In particular, the spectral and the nuclear norms are tight.

Proof.

We have

[TABLE]

In particular, $\mu_{XY}(C^{(j)})$ is strictly decreasing as $j$ increases.

If $i\leq j$ then we have

[TABLE]

This proves that (7) is the slope decomposition. ∎

7.3. Principal Component Analysis

In Principal Component Analysis (PCA), one finds a low rank matrix that approximates the matrix $A$ by truncating the singular value decomposition. For a given threshold $y$, let $s$ be maximal such that $\lambda_{s}>y$. Then

[TABLE]

is a low rank approximation of $C$. This method is called hard threshholding. Replacing $C$ by the approximation $C^{\prime}$ is an effective way to reduce the dimension of a large scale problem. Let us compare this to the $XY$-decomposition (or equivalently $X2$- or $2Y$-decomposition) of $C$. Let us define

[TABLE]

and

[TABLE]

Lemma 7.3**.**

The expression $C=A+B$ is an $XY$-decomposition.

Proof.

We have $\|A\|_{*}=\|A\|_{X}=\sum_{i=1}^{s}m_{i}(\lambda_{i}-y)$ and $\|B\|_{\sigma}=\|B\|_{Y}=y$. Now the lemma follows from

[TABLE]

∎

In particular, $A=\operatorname{shrink}_{Y}(C,y)$. The operator $\operatorname{shrink}_{Y}(\cdot,y)$ is soft-threshholding with threshold level $y$ (see [20]). Unlike hard thresholding, soft thresholding is continuous. The Pareto frontier (which is also the sub-frontier) is given by

[TABLE]

7.4. The singular value region for matrices

The Pareto frontier of $C$ encodes the singular values. We will describe in detail how to obtain the singular values from the Pareto frontier. Note that we consider $x=h_{XY}(y)$ as a function of $y$, rather then considering its inverse function $y=h_{YX}(x)$. If we differentiate $h_{XY}^{C}(y)$ with respect to $y$ we get

[TABLE]

if $\lambda_{s}>y>\lambda_{s+1}$ and $1\leq s\leq r$ (with the convention that $\lambda_{r+1}=0$).

If we plot $y$ against $x=-(h_{XY}^{C}(y))^{\prime}$ then we get the singular value region. From the descriptions above, it is now clear that this region can be described as an $m_{1}\times\lambda_{1}$ bar, followed by an $m_{2}\times\lambda_{2}$ bar etc. So the singular value region from Definition 2.20 is the same as the singular value region for matrices as described in the introduction.

7.5. Rank minimization and low rank matrix completion

Let $V={\mathbb{C}}^{m\times n}$ be the set of $m\times n$ matrices. We will study the low rank matrix completion from the viewpoint of competing dual norms.

Problem 7.4** (Low Rank Matrix Completion (LRMC)).**

Given a $m\times n$ matrix $C$ where the entries $(i_{1},j_{1}),\dots,(i_{s},j_{s})$ are missing. Fill in the missing entries such that the resulting matrix has minimal rank.

The low rank matrix completion problem has applications in collaborative filtering and recommender systems such as the Netflix problem. The low rank matrix completion problem is a special case of the rank minimization problem.

Problem 7.5** (Rank Minimization (RM)).**

Suppose that $W$ is a subspace of $V$, and let $A\in V$. Find a matrix $B\in A+W$ of minimal rank.

The Low Rank Matrix Completion problem can be formulated as a rank minimization problem as follows. Complete $C$ to a matrix $A$ in some way (for example, set all the missing entries equal to [math]). Then, Let $W\subseteq V$ be the subspace spanned by all matrices $e_{i_{k},j_{k}}$, $k=1,2,\dots,s$. Here $e_{p,q}$ is the matrix with all [math]’s except for a 1 in position $(p,q)$. Find $B\in A+W$ with minimal rank using RM. Then $B$ is also the solution to the LRMC problem.

Let us consider the Rank Minimization problem. Using the philosophy of convex relaxation, we consider the following problem instead (see [12, 15, 45]):

Problem 7.6**.**

Find a matrix $D\in C+W$ with $\|D\|_{\star}$ minimal.

Let $Z$ be the orthogonal complement of $W$ and let $\pi_{Z}$ be the orthogonal projection onto $Z$. The problem does not change when we replace $C$ by $\pi_{Z}(C)$, so we may assume that $C\in Z$ without loss of generality. We define a norm $\|\cdot\|_{X}$ on $Z$ by

[TABLE]

So Problem 7.6 is essentially the problem of determining the value of $\|C\|_{X}$. In the presence of noise, we would like to find a matrix $A$ such that $\|A\|_{X}$ and $\|C-A\|_{2}$ are small. This leads to the following optimization problem.

Problem 7.7**.**

For a fixed parameter $\lambda$, minimize

[TABLE]

We can write $A=\pi_{Z}(D)$ such that $\|A\|_{X}=\|D\|_{\star}$. We can reformulate the problem as:

Problem 7.8**.**

For a fixed parameter $\lambda$, minimize

[TABLE]

The dual norm to $\|\cdot\|_{X}$ is defined by

[TABLE]

Problem 7.7 is equivalent to

Problem 7.9**.**

Minimize $\|A\|_{2}$ under the constraint $\|C-\pi_{Z}(A)\|_{\sigma}\leq\lambda$.

8. Restricting Norms

Suppose that $V$ is a finite dimensional ${\mathbb{R}}$-vector space with a positive definite bilinear form $\langle\cdot,\cdot\rangle$, and $\|\cdot\|_{X}$ is a norm on $V$. For a subspace $W$ of $V$, it is natural to ask whether the $X2$-decompositions of vectors in $W$ are always within the space $W$. In this section we will give a sufficient criterion for $W$ to have this property.

Definition 8.1**.**

A subspace $W\subseteq V$ is called a nice slice if we have $\|\pi_{W}(c)\|_{X}\leq\|c\|_{X}$ for all $c\in V$, where $\pi_{W}:V\to W$ is the orthogonal projection.

Lemma 8.2**.**

If $W$ is a nice slice, then we also have $\|\pi_{W}(c)\|_{Y}\leq\|c\|_{Y}$ for all $c\in V$, where $\|\cdot\|_{Y}$ is the norm dual to $\|\cdot\|_{X}$.

Proof.

Choose a vector $d$ with $\|d\|_{X}=1$ and $\langle d,\pi_{W}(c)\rangle=\|\pi_{W}(c)\|_{Y}$. We have

[TABLE]

∎

Let $\operatorname{O}(V)$ be the orthogonal group consisting of all $g\in\operatorname{GL}(V)$ with the property

[TABLE]

for all $v,w\in V$.

Lemma 8.3**.**

Suppose that $G\subseteq\operatorname{O}(V)$ is a subgroup with the properties $\|g\cdot v\|_{X}=\|v\|_{X}$ for all $v,w\in V$ and all $g\in G$. Then the space $V^{G}=\{v\in V\mid\forall g\in G\ g\cdot v=v\}$ of $G$-invariant vectors is a nice slice.

Proof.

We can replace $G$ with its closure, so without loss of generality we may assume that $G$ is a compact Lie group. Let $\pi_{W}$ be the projection onto $W:=V^{G}$. We have

[TABLE]

where $d\mu$ is the normalized Haar measure. This shows that $\pi_{W}(v)$ lies in the convex hull of all $g\cdot v$, $g\in G$. Since $\|g\cdot v\|_{X}\leq\|v\|_{X}$ for all $v\in V$, we also have $\|\pi_{W}(v)\|_{X}\leq\|v\|_{X}$ for all $g\in G$. ∎

Suppose that $W\subseteq V$ is a nice slice. Let $\|\cdot\|_{X^{\prime}}$ and $\|\cdot\|_{Y^{\prime}}$ be the restrictions of the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ to $W$.

Lemma 8.4**.**

The norms $\|\cdot\|_{X^{\prime}}$ and $\|\cdot\|_{Y^{\prime}}$ are also dual to each other.

Proof.

If $v,u\in W$ and $\|u\|_{Y}=1$ then we have

[TABLE]

For a given $v\in W$, there exists $w\in V$ with $\|w\|_{Y}=1$ and

[TABLE]

We get

[TABLE]

Define $u=\pi_{W}(w)/\|\pi_{W}(w)\|_{Y^{\prime}}$. We have $\|u\|_{Y^{\prime}}=1$. It follows that

[TABLE]

because $\|\pi_{W}(w)\|_{Y^{\prime}}=\|\pi_{W}(w)\|_{Y}\leq\|w\|_{Y}=1$. So $\langle v,u\rangle=\|v\|_{X^{\prime}}$. ∎

Lemma 8.5**.**

Suppose that $c\in W$.

1. (1)

If $c=a+b$ is an $X2$-decomposition, then $a,b\in W$ and $c=a+b$ is an $X^{\prime}2$ decomposition. 2. (2)

If $c=a+b$ is an $XY$ decomposition then $c=\pi_{W}(a)+\pi_{W}(b)$ is an $X^{\prime}Y^{\prime}$ decomposition, $\|\pi_{W}(a)\|_{X^{\prime}}=\|a\|_{X}$ and $\|\pi_{W}(b)\|_{Y^{\prime}}=\|b\|_{Y}$.

Proof.

(1) If $c=a+b$ is an $X2$ demposition, then we have

[TABLE]

$\|\pi_{W}(a)\|_{X}\leq\|a\|_{X}$ and $\|\pi_{W}(b)\|_{2}\leq\|b\|_{2}$. It follows that $c=\pi_{W}(a)+\pi_{W}(b)$ is also an $X2$-decomposition and by uniqueness we have $\pi_{W}(a)=a$ and $\pi_{W}(b)=b$. Suppose that $c=a^{\prime}+b^{\prime}$ with $a^{\prime},b^{\prime}\in W$ and $\|b^{\prime}\|_{2}=\|b\|_{2}$. We get

[TABLE]

because $c=a+b$ is an $X2$-decomposition. This shows that $c=a+b$ is an $X^{\prime}2$-decomposition.

(2) Suppose that $c=a+b$ is an $XY$-decomposition. We get

[TABLE]

$\|\pi_{W}(a)\|_{X}\leq\|a\|_{X}$ and $\|\pi_{W}(b)\|_{Y}\leq\|b\|_{Y}$. It follows that $c=\pi_{W}(a)+\pi_{W}(b)$ is also an $XY$ decomposition. A similar argument as in (1) shows that this is also an $X^{\prime}Y^{\prime}$-decomposition. Since $\|\pi_{W}(a)\|_{X}\leq\|a\|_{X}$, we must have $\|b\|_{Y}\geq\|\pi_{W}(b)\|_{Y}\geq\|b\|_{Y}$, because $c=a+b$ is an $XY$-decomposition. It follows that

[TABLE]

Similarly, we get $\|\pi_{W}(a)\|_{X^{\prime}}=\|a\|_{X}$. ∎

In particular, we have $f^{c}_{YX}=f^{c}_{Y^{\prime}X^{\prime}}$ and $h^{c}_{YX}=h^{c}_{Y^{\prime}X^{\prime}}$.

Example 8.6**.**

Suppose that $V={\mathbb{C}}^{m\times n}$ and $W={\mathbb{R}}^{m\times n}\subseteq V$. The orthogonal projection $\pi_{W}:V\to W$ is given by $\pi_{W}(A)=\Re(A)$. Suppose that $A=A_{1}+A_{2}i\in{\mathbb{C}}^{m\times n}$ where $A_{1},A_{2}\in{\mathbb{R}}^{m\times n}$. Choose a unit vector $v\in{\mathbb{R}}^{n}$ such that $\|\pi_{W}(A)v\|_{2}=\|\pi_{W}(A)\|_{\sigma}$. Then we have

[TABLE]

This shows that $W$ is a nice slice.

9. 1D total variation denoising

9.1. The total variation norm and its dual

In this section we discuss the application to 1-dimensional total variation denoising. This example is particularly interesting because the corresponding norms are tight.

Define the difference map $D:{\mathbb{R}}^{n+1}\to{\mathbb{R}}^{n}$ by

[TABLE]

The map $D$ is surjective, and the kernel is spanned by the vector ${\bf 1}=(1\ 1\cdots 1)^{t}$. The dual map $D^{\star}:{\mathbb{R}}^{n}\to{\mathbb{R}}^{n+1}$ is given by

[TABLE]

If we compose the two maps we get

[TABLE]

The linear map $DD^{\star}$ is invertible. Let $V\subset{\mathbb{R}}^{n+1}$ be the subspace defined by

[TABLE]

The image of $D^{\star}$ is $V$ If $b=D^{\star}(DD^{\star})^{-1}a$, then $b$ is the vector of minimal length with the property $Db=a$.

We define a norm $\|\cdot\|_{X}$ on ${\mathbb{R}}^{n}$ by

[TABLE]

Another norm $\|\cdot\|_{Y}$ on ${\mathbb{R}}^{n}$ is given by

[TABLE]

Lemma 9.1**.**

Suppose that $b$ is a vector with $Db=a$. Let $m_{+}(b)=\max\{b_{1},\dots,b_{n+1}\}$ and $m_{-}(b)=\min\{b_{1},\dots,b_{n+1}\}$. Then we have $\|a\|_{Y}=\frac{1}{2}(m_{+}(b)-m_{-}(b))$.

Proof.

The vectors that map to $a$ under $D$ are of the form $b-\lambda{\bf 1}$. We have $\|b-\lambda{\bf 1}\|_{\infty}=\max\{m_{+}-\lambda,m_{-}-\lambda\}$. This quantity is minimal if $\lambda=\frac{1}{2}(m_{+}(b)+m_{-}(b))$. In that case we have $\|b-\lambda{\bf 1}\|_{\infty}=\frac{1}{2}(m_{+}(b)-m_{-}(b))$. ∎

Lemma 9.2**.**

The norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are dual to each other.

Proof.

Suppose that $a,b\in{\mathbb{R}}^{n}$. Choose $e\in{\mathbb{R}}^{n+1}$ such that $De=b$ and $\|e\|_{\infty}=\|b\|_{Y}$. Then we have

[TABLE]

Suppose that $a$ is nonzero and let $c=D^{\star}a\neq 0$. Define

[TABLE]

Because $c_{1}+\cdots+c_{n+1}=0$, the set $\{c_{1},\dots,c_{n+1}\}$ has positive and negative elements. This implies that $m_{+}(e)=1$ and $m_{-}(e)=-1$. If $b=De$ then we have $\|b\|_{Y}=\frac{1}{2}(m_{+}(e)-m_{-}(e))=1$.

[TABLE]

This shows that the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are dual. ∎

For a vector $a\in{\mathbb{R}}^{n}$, $\|D^{\star}a\|_{1}$ is its total variation. Given a signal $c$ and an $\varepsilon>0$, a solution $a$ to the problem ${\bf M}_{X2}(\varepsilon)$ minimizes the total variation $\|D^{\star}a\|_{1}$ under the constraint $\|c-a\|_{2}\leq\varepsilon$. The function $a$ is typical a piecewise constant function. Below is an example, where the blue function is $c$ and the red function is $a$.

As we increase the value of $\varepsilon$, the sparsity decreases. Below we draw the signal $c$ in blue, and (a vertical translation of) the denoised signal $a=\operatorname{shrink}_{Y}(\varepsilon)$ for various values of $\varepsilon$ in red.

9.2. Description of the unit ball

We now will describe the unit balls $B_{X}$ and $B_{Y}$.

Definition 9.3**.**

A vector $(s_{1}\ s_{2}\ \cdots\ s_{n+1})^{t}$ is called a signature sequence if $s_{1},s_{2},\dots,s_{n+1}\in\{-1,0,1\}$ and $\{1,-1\}\subseteq\{s_{1},\dots,s_{n+1}\}$. For a signature sequence $s$, we define $\widetilde{F}_{s}$ as the set of all vectors $x\in{\mathbb{R}}^{n+1}$ such that $x_{i}=s_{i}$ when $s_{i}=\pm 1$, and $|x_{i}|\leq 1$ if $s_{i}=0$. We define $F_{s}=D(\widetilde{F}_{s})$.

Lemma 9.4**.**

The set $F_{s}$ is a face of the unit ball $B_{Y}$.

Proof.

The set $\widetilde{F}_{s}$ is a face of the unit ball $B_{1}\subset{\mathbb{R}}^{n+1}$ for the $\ell_{1}$ norm. Suppose that $b=tb_{1}+(1-t)b_{2}\in F_{s}$ with $b_{1},b_{2}\in B_{Y}$ and $0<t<1$. Then we can write $b_{i}=D(\widetilde{b}_{i})$ with $\|\widetilde{b}_{i}\|_{\infty}\leq 1$ for $i=1,2$. We have

[TABLE]

for some $\widetilde{b}\in\widetilde{F}_{s}$. We get that $\widetilde{b}$ and $t\widetilde{b}_{1}+(1-t)\widetilde{b}_{2}$ differ by a multiple of ${\bf 1}$. Since the maximum and minimum entry of $\widetilde{b}$ are $1$ and $-1$ respectively, and $\|t\widetilde{b}_{1}+(1-t)\widetilde{b}_{2}\|_{\infty}\leq 1$ we deduce that $\widetilde{b}=t\widetilde{b}_{1}+(1-t)\widetilde{b}_{2}$. Now $\widetilde{b}_{1},\widetilde{b}_{2}$ lie in the unit ball $B_{1}$ and $\widetilde{b}$ lies in the face $\widetilde{F}_{s}$. It follows that $\widetilde{b}_{1},\widetilde{b}_{2}\in\widetilde{F}_{s}$ and $b_{1}=D(\widetilde{b}_{1}),b_{2}=D(\widetilde{b}_{2})\in F_{s}$. ∎

The dimension of $F_{s}$ is equal to the number of [math]’s in the signature sequence $s$. The restriction of $D$ to $\widetilde{F}_{s}$ is injective, so $F_{s}$ and $\widetilde{F}_{s}$ have the same dimension.

Proposition 9.5**.**

The faces of $B_{Y}$ of dimension $<n$ are exactly all $F_{s}$ where $s$ is a signature sequence.

Proof.

Suppose that $F$ is a proper face of the polytope $B_{Y}$. Then there exists a vector $a\in{\mathbb{R}}^{n}$ with $\|a\|_{X}=1$ and $F=\{b\in B_{Y}\mid\langle a,b\rangle=1\}$. Let $\widetilde{a}=D^{\star}a$ and $s_{i}=\operatorname{sgn}(\widetilde{a}_{i})$ for $i=1,2,\dots,n+1$. Since $\widetilde{a}\neq 0$, and $\widetilde{a}_{1}+\cdots+\widetilde{a}_{n+1}=0$, the vector $\widetilde{a}$ must have positive and negative coordinates. In the sequence $s_{1},\dots,s_{n+1}$ the elements $1$ and $-1$ both must appear. For a vector $b\in{\mathbb{R}}^{n}$ with $\|b\|_{Y}=1$, let $\widetilde{b}\in{\mathbb{R}}^{n+1}$ be the unique vector with $D\widetilde{b}=b$ and $\|\widetilde{b}\|_{\infty}=1$. Then we have

[TABLE]

with $\|\widetilde{a}\|_{1}=\|\widetilde{b}\|_{\infty}=1$. Now $\langle\widetilde{a},\widetilde{b}\rangle=\sum_{i=1}^{n+1}\widetilde{a}_{i}\widetilde{b}_{i}=1$ if and only if for every $i$, $\widetilde{a}_{i}=s_{i}=0$ or $\widetilde{b}_{i}=s_{i}$. In other words, $\langle\widetilde{a},\widetilde{b}\rangle=1$ if and only if $\widetilde{b}\in\widetilde{F}_{s}$. So $\langle a,b\rangle=\langle\widetilde{a},\widetilde{b}\rangle=1$ if and only if $b=D\widetilde{b}\in F_{s}$. ∎

Theorem 9.6**.**

The norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are tight.

Proof.

Suppose that $s$ is a signature vector. It suffices to construct a unitangent vector $u\in F_{s}$ by Proposition 6.12. We define a vector $v\in{\mathbb{R}}^{n+1}$ as follows. If $s_{i}=\pm 1$ then $v_{i}=s_{i}$. The other coordinates of $v$ are obtained by linear interpolation. If $i<j$, $s_{i},s_{j}\neq 0$ and $s_{i+1}=\cdots=s_{j-1}=0$ then we define

[TABLE]

whenever $i<k<j$. If $s_{1}=s_{2}=\cdots=s_{i-1}=0$ and $s_{i}\neq 0$ then we define $v_{k}=s_{i}$ for $k<i$. If $s_{j+1}=s_{j+2}=\cdots=s_{n+1}=0$ and $s_{j}\neq 0$ then we define $v_{k}=s_{j}$ for $k>j$. For example, if $s=(0,0,1,0,0,0,-1,0,-1,0,0,1)$ then $v=(1,1,1,\frac{1}{2},0,-\frac{1}{2},-1,-1,-1,-\frac{1}{3},\frac{1}{3},1)$.

From the construction follows that for every $i$ we have $(D^{\star}Dv)_{i}s_{i}=|(D^{\star}Dv)_{i}|$. We define $u=Dv$. We have $\|u\|_{Y}=\|v\|_{\infty}=1$. From the construction it is clear that $v\in\widetilde{F}_{s}$ and $u\in F_{s}$. We have

[TABLE]

so $u$ is unitangent. ∎

We briefly discuss the combinatorics of the polytope $B_{Y}$. Let $h_{i}$ be the number of faces of $B_{Y}$ of dimension $i$. For $i<n$, $h_{i}$ is the number of signature sequences with $i$ zeroes. We also have $h_{n}=1$. The generating function for $h_{0},h_{1},\dots,h_{n}$ is $H(t)=h_{0}+h_{1}t+\cdots+h_{n}t^{n}$. The generating function for the set $\{1,0,-1\}^{n+1}$ is $(2+t)^{n+1}$. The generating function for $\{1,0\}^{n+1}$ and for $\{-1,0\}^{n+1}$ is $(1+t)^{n+1}$. The generating function for $\{(0,0,\dots,0)\}$ is $t^{n+1}$. So the generating function for the set of signature sequences, using inclusion-exclusion, is $(2+t)^{n+1}-2(1+t)^{n+1}+t^{n+1}$. There is one face of dimension $n$ that does not correspond to a signature sequences, so we have

[TABLE]

In particular, $h_{0}=2^{n+1}-2$ is the number of vertices of the ball $B_{Y}$, and $h_{n-1}=4{n+1\choose 2}-2{n+1\choose 2}=n^{2}+n$ is the number of facets of $B_{Y}$, which is the number of vertices of $B_{X}$. The total number of faces of $B_{Y}$ (and $B_{X}$) is $H(1)=3^{n+1}-2^{n+2}+2$.

Example 9.7**.**

Let $n=3$. We have the following signature sequences and corresponding unitangent vectors (written as row vectors):

[TABLE]

Every vertex of the unit ball $B_{Y}$ is unitangent. These vectors are marked with $\star$ and they correspond to signature sequences that have no zeroes. Below is a 2-dimensional projection of the unit ball $B_{Y}$.

The unit ball $B_{X}$ is dual to the polytope $B_{Y}$ and is shown below:

An example of an $XY$-slope decomposition is:

[TABLE]

If $a\in{\mathbb{R}}^{n}$, then we have

[TABLE]

9.3. The taut string method

The restriction of $D:{\mathbb{R}}^{n+1}\to{\mathbb{R}}^{n}$ to $V$ gives an isomorphism between $V$ and ${\mathbb{R}}^{n}$. Now we can view the bilinear form $\langle\cdot,\cdot\rangle$ as a bilinear form on $V$ and for $a,b\in V$ we have

[TABLE]

In particular, we have

[TABLE]

We can also view $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ as norms on $V$ and for $c=(c_{1},\dots,c_{n+1})\in V$ we have

[TABLE]

and

[TABLE]

For $c\in{\mathbb{R}}^{n+1}$, we consider the following optimization problem:

${\bf TS}^{c}(\varepsilon)$: Find a vector $a\in{\mathbb{R}}^{n+1}$ with $\|c-a\|_{\infty}\leq\varepsilon$ such that $\|D^{\star}Da\|_{1}$ is minimal.

We call this the Taut String problem. This problem, and some generalizations to higher order, were studied in [39]. If the vectors $a,b,c$ are discretized functions, Then the graph $a$ lies between $c-\varepsilon$ and $c+\varepsilon$. Now $D^{\star}Da$ is a discrete version of the second derivative. The value $|D^{\star}Da)_{i}|$ is a measure of how much the graph bends at vertex $i$. So $\|D^{\star}Da\|_{1}$ is the total amount of bending and we try to minimize this. Visually we can see $a$ as a string between $c-\varepsilon$ and $c+\varepsilon$ and we pull $a$ on both ends so that the string is taut. The Taut String Algorithm described in [18] computes $a$ in time $O(n)$ (see also [11]). The function $a$ is piecewise linear and $D(a)$ is piecewise constant. Total Variation Denoising of time signals has applications in statistics to estimate a density function from a collection of measurements (see [3]). It was also used in [5, 40] for analysing heart rate variabilty signals to predict hemodynamic decompensation. In the graph below, we have drawn $c-\varepsilon$ and $c+\varepsilon$ for a function $c$, as well as the solution $a$ to the Taut String problem ${\bf TS}^{c}(\varepsilon)$. It appears as a tight string that is in between the graphs of $c-\varepsilon$ and $c+\varepsilon$ and is a piecewise linear approximation of $c$.

As the value of $\varepsilon$ increases, the sparsity decreases:

Let $\pi:{\mathbb{R}}^{n+1}\to V$ be the projection onto $V$ defined by

[TABLE]

Lemma 9.8**.**

Suppose that $c\in V$ and $a\in{\mathbb{R}}^{n+1}$ minimizes $\|D^{\star}Da\|_{1}$ under the constraint $\|c-a\|_{\infty}\leq\varepsilon$. Then $\pi(a)$ is a solution to ${\bf M}_{XY}^{c}(\varepsilon)$ and $c=\pi(a)+\pi(b)$ is an $XY$-decomposition, where $b=c-a$.

Proof.

We have $\|\pi(b)\|_{Y}\leq\|b\|_{\infty}\leq\varepsilon$. Suppose that $c=a^{\prime}+b^{\prime}$ with $a^{\prime},b^{\prime}\in V$ and $\|b^{\prime}\|_{Y}\leq\varepsilon$. Then there is a vector $\overline{b}\in{\mathbb{R}}^{n+1}$ with $\pi(\overline{b})=b$ and $\|\overline{b}\|_{\infty}=\|b^{\prime}\|_{X}\leq\varepsilon$. If we define $\overline{a}=c-\overline{b}$, then $\pi(\overline{a})=a^{\prime}$ and we have

[TABLE]

This shows that $\pi(a)$ is a solution to ${\bf M}_{XY}^{c}(\varepsilon)$. ∎

9.4. An ECG example

For a $500$ Hz noisy electrocardiogram (ECG) signal $c\in{\mathbb{R}}^{2000}$ of 4 seconds long, we graph the Pareto frontier (and sub-frontier) of $c$:

(If $c\not\in V$, then we can remove the baseline by replacing $c$ with $\pi(c)\in V$.) The graph is $L$-shaped, where the vertical leg corresponds to the sparse signal, and the horizontal leg corresponds to noise. The vertical leg starts near the point $(300,8)$. This means that there exists a decomposition $c=a+b$ with $\|b\|_{\infty}=8$ and $\|D^{\star}Da\|_{1}\approx 300$. The signal $a$ is the denoised signal. The singular value region for $c$ is shown below:

The $x$-axis is cut of here in order to better visualize the graph. The graph approaches the $x$-axes slowly and meets the $x$-axis near $x=1000$. The horizontal leg corresponds to noise. To estimate the noise level, we look at where the horizontal leg starts. To find a cutoff for the singular values one proceeds as in principal component analysis. A reasonable cutoff is again $y=8$. The value 8 is an estimation of the maximum amplitude of noise, which is more than the standard deviation (which is closer to $4$ in this case). Below we graph the noisy signal, and the denoised signals for $y=2,4,6,8,10,12$. The denoised signal for $y=8$ is colored green.

9.5. higher order total variation denoising

Suppose that $c:[0,1]\times[0,1]\to{\mathbb{R}}$ is a function. One can also denoise by using a higher derivative to regularize the $\ell_{2}$ norm by minimizing

[TABLE]

is small. For $k=1$ we just get 1D Total Variation Denoising. We consider a discrete version.

Define $D^{(k)}:{\mathbb{R}}^{n+k}\to{\mathbb{R}}^{n}$ as the composition $\underbrace{D\circ D\circ\cdots\circ D}_{k}$. We can view $D^{(k)}$ as the $k$-th discrete derivative. For example, for $k=2$ we have

[TABLE]

Define a semi-norm on ${\mathbb{R}}^{n+k}$ by $\|v\|_{X}=\|D^{(k)}v\|_{1}$. Restricting the norm to

[TABLE]

gives a norm, and let $\|\cdot\|_{Y}$ be the dual to this norm.

Problem 9.9**.**

Given $c\in{\mathbb{R}}^{n+k}$, minimize $\frac{1}{2}\|c-a\|_{2}^{2}+\lambda\|a\|_{X}$.

For $k=2$, this problem is called $\ell_{1}$-trend filtering. An overview of $\ell_{1}$-trend filtering is given in [33]. Some applications are in financial time series ([53]), macroeconomics ([52]) automatic control ([41]), oceanography ([51]) and geophysics ( [34]). In $\ell_{1}$-trend filtering, the function $a$ is piecewise linear. More generally, for $k>2$, the $(k-1)$-th derivative of $a$ will be piecewise constant. This case has been studied in [39]. For $k\geq 2$, the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are not tight.

10. The ISTA algorithm for $XY$-decompositions

A general formulation of the Iterative Shrinkage-Tresholding Algorithm (ISTA) was given in [17]. A map $f:{\mathbb{R}}^{n}\to{\mathbb{R}}^{n}$ is called non-expansive if it is Lipschitz with Lipschitz constant 1, i.e., $\|f(v)-f(w)\|_{2}\leq\|v-w\|_{2}$ for all $v,w\in{\mathbb{R}}^{n}$.

Lemma 10.1**.**

If $\|\cdot\|_{X}$ is a norm on ${\mathbb{R}}^{n}$, then the functions $\operatorname{proj}_{X}(\cdot,x)$ and $\operatorname{shrink}_{X}(\cdot,x)$ are non-expansive.

Proof.

Suppose that $v,w\in{\mathbb{R}}^{n}$, and let $v^{\prime}=\operatorname{proj}_{X}(v,x)$ and $w^{\prime}=\operatorname{proj}_{X}(w,x)$. For $t\in[0,1]$ we have $(1-t)v^{\prime}+tw^{\prime}\in B_{X}(x)$. By definition of $v^{\prime}=\operatorname{proj}_{X}(v,x)$ we have

[TABLE]

Squaring both sides yields

[TABLE]

If we take the limit $t\to 0$, we get

[TABLE]

By symmetry, we also get

[TABLE]

Adding both equations, yields

[TABLE]

It follows that

[TABLE]

This shows that $\operatorname{proj}_{X}(\cdot,x)$ and $\operatorname{shrink}_{X}(\cdot,x)$ are nonexpansive.

∎

Suppose that $V={\mathbb{R}}^{n}$. Let $D:{\mathbb{R}}^{m}\to{\mathbb{R}}^{n}$ be a surjective linear map and suppose that $\|\cdot\|_{\overline{X}}$ is a norm on ${\mathbb{R}}^{m}$. We define a norm $\|\cdot\|_{X}$ on ${\mathbb{R}}^{n}$ by

[TABLE]

The dual norm $\|\cdot\|_{Y}$ of $\|\cdot\|_{X}$ is defined by

[TABLE]

where $\|\cdot\|_{\overline{Y}}$ is the dual norm to $\|\cdot\|_{\overline{X}}$. Assume that we can easily compute the norms $\|\cdot\|_{\overline{X}}$, $\|\cdot\|_{\overline{Y}}$ and the projection function $\operatorname{proj}_{\overline{X}}$. We will also assume that the singular values of $D$ all lie in $[0,1]$. We have the following algorithm for computing $\operatorname{proj}_{X}(c,x)$ for $c\in V$ and $x\geq 0$. There is one more parameter, $\delta$, which specifies the accuracy of the output. We assume $0<\delta<1$ and the closer $\delta$ is to $1$, the more accurate the output will be.

1:function ${\rm proj}_{X}$($c,x,\delta$)

2:     $e\leftarrow 0$

3:     while  $\langle De,c-De\rangle\leq\delta\cdot\|e\|_{\overline{X}}\cdot\|D^{\star}(c-De)\|_{\overline{Y}}$  do

4:         $e\leftarrow\operatorname{proj}_{\overline{X}}(2D^{\star}(c-De))+e,x)$

5:     end while

6:     return $De$

7:end function

11. Basis Pursuit Denoising, LASSO and the Dantzig selector

In Basis Pursuit (BP) one tries to solve the equation $Av=c$ where $A$ is a given $n\times m$ matrix and $v$ is a sparse vector (few nonzero entries). We will assume that $A$ has rank $n$ and that the system is underdetermined, ($m>n$). It was shown in [13] that $v$ often can be found by minimizing $\|v\|_{1}$ under the constraint $Av=c$. This can be done efficiently using linear programming. We define a norm $\|\cdot\|_{X}$ on ${\mathbb{R}}^{n}$ by

[TABLE]

Note that evaluating the norm $\|c\|_{X}$ of some vector $c$ is a Basis Pursuit problem. We also have

[TABLE]

In the presence of noise, one minimizes $\|v\|_{1}$ under the constraint $\|Av-c\|_{2}\leq y$. This is called Basis Pursuit Denoising (BPDN), see [9, 16]. If we set $a=Av$, then we minimize $\|a\|_{X}$ under the the constraint $\|c-a\|_{2}\leq y$. If we set $b=c-a$, then we minimize $\|c-b\|_{X}$ under the constraint $\|b\|_{2}\leq y$. This is the optimization problem ${\bf M}_{X2}^{c}(y)$.

Sometimes BPDN is formulated as the problem of minimizing $\ell_{1}$-regularized function

[TABLE]

This is the same problem as minimizing

[TABLE]

The equivalence between the two formulations of BPDN is well-known (see also Proposition 4.6).

The LASSO problem asks to minimize $\|c-Av\|_{2}$ under the constraint $\|v\|_{1}\leq x$. This is equivalent to minimizing $\|c-a\|_{2}$ under the constraint $\|a\|_{X}\leq x$. This is the optimization problem ${\bf M}_{2X}^{c}(x)$.

The dual norm of $\|\cdot\|_{X}$ is defined by

[TABLE]

Since the $X2$-decompositions and the $2Y$ decompositions are the same, we have two more approaches for finding the $X2$-decompositions (dual LASSO/BPDN):

1. (1)

${\bf M}^{c}_{2Y}(y)$: Under the constraint $\|A^{\star}(c-a)\|_{\infty}\leq y$ we minimize $\|a\|_{2}$ ([43, 6]). 2. (2)

${\bf M}^{c}_{Y2}(x)$: Under the constraint $\|a\|_{2}\leq x$, we minimize $\|A^{\star}(c-a)\|_{\infty}$.

To find an $XY$-decomposition we can minimize $\|a\|_{X}$ under the constraint $\|c-a\|_{Y}\leq y$ (${\bf M}^{c}_{XY}(y)$). If we set $a=Av$ then this is equivalent to minimizing $\|v\|_{1}$ under the constraint $\|A^{\star}(Av-c)\|_{\infty}\leq y$. This optimization problem is called the Dantzig selector ([14]).

The Dantzig selector does not always have the same solution as LASSO (or the other equivalent problems). Some conditions were given in [1] when the Dantzig selector and LASSO have the same solutions. If $a_{1},a_{2},\dots,a_{m}$ are the columns of $A$, then the unit ball $B_{X}$ is the convex hull of $a_{1},\dots,a_{m},-a_{1},\dots,-a_{m}$. Theorem 6.12 gives a necessary and sufficient condition for this polytope so that Dantig selector and LASSO have the same solutions for every $c\in V$.

12. Total Variation Denoising in Imaging

12.1. The 2D total variation norm

We can view a grayscale image as a function $c:[0,1]\times[0,1]\to{\mathbb{R}}$. In the anisotropic Rudin-Osher-Fatemi ([46]) total variation model, we seek a decomposition $c=a+b$ such that

[TABLE]

is small. (In the isotropic model we replace $\|\nabla a(x,y)\|_{1}$ by $\|\nabla a(x,y)\|_{2}$.)

A discrete formulation of the model is as follows. We can view a grayscale image of $m\times n$ pixels as a matrix $c\in{\mathbb{R}}^{m\times n}$. We define a map $F:{\mathbb{R}}^{m\times n}\to{\mathbb{R}}^{m\times(n-1)}\times{\mathbb{R}}^{(m-1)\times n}$ by

[TABLE]

where $d_{i,j}=c_{i,j}-c_{i,j+1}$ and $d_{i,j}=e_{i,j}-e_{i,j+1}$ for all $i,j$. We define a total variation semi-norm by

[TABLE]

The restriction of $\|\cdot\|_{X}$ to the set

[TABLE]

is a norm. We can always normalize an image by subtracting the average value to obtain an element of $V$. Let $F^{\star}:{\mathbb{R}}^{m\times(n-1)}\times{\mathbb{R}}^{(m-1)\times n}\to{\mathbb{R}}^{m\times n}$ be the dual of $F$. We have

[TABLE]

with the conventions that $d_{i,0}=d_{i,n}=e_{0,j}=e_{m,j}=0$ for all $i,j$.

The dual norm of $\|\cdot\|_{X}$ is

[TABLE]

Total Variation Denoising is usually formulated as follows

Problem 12.1**.**

Minimize

[TABLE]

is minimal.

Since the paper of Rudin, Osher and Fatemi ([46]), several other algorithms have been proposed, for example Chambolle’s algorithm ([10]), the split Bregman method ([24]) and the efficient primal-dual hybrid gradient algorithm ([54]).

A region in $R\subseteq\{1,2,\dots,m\}\times\{1,2,\dots,n\}$

12.2. Spareseness and total variation

We will study the notion of $X$-sparsity in this context. Suppose that $F\in V$ is an image. An $F$-region is a maximal connected subset of $\{1,2,\dots,m\}\times\{1,2,\dots,n\}$ on which $F$ is constant.

Proposition 12.2**.**

For an image $F$ we have $\operatorname{gsparse}(F)=d-1$ where $d$ is the number of connected regions on which $F$ is constant.

Proof.

Let $C$ be the smallest facial $X$-cone containing $F$. By Lemma 6.6, $F$ lies in $C$ if and only if $\|F-\varepsilon G\|_{X}+\|\varepsilon G\|_{X}=\|F\|_{X}$ for some $\varepsilon>0$. So $G$ lies in $C$ if and only if the following properties are satisfied for all $i,j$:

1. (1)

$G(i,j)>G(i,j+1)$ implies $F(i,j)>F(i,j+1)$; 2. (2)

$G(i,j)<G(i,j+1)$ implies $F(i,j)<F(i,j+1)$; 3. (3)

$G(i,j)>G(i+1,j)$ implies $F(i,j)>F(i+1,j)$; 4. (4)

$G(i,j)<G(i+1,j)$ implies $F(i,j)<F(i+1,j)$.

Taking the contrapositive in each statement (and changing the indexing), we see that for all $i,j$ we have:

1. (1)

$F(i,j)\geq F(i,j+1)$ implies $G(i,j)\geq G(i,j+1)$; 2. (2)

$F(i,j)\leq F(i,j+1)$ implies $G(i,j)\leq G(i,j+1)$; 3. (3)

$F(i,j)\geq F(i+1,j)$ implies $G(i,j)\geq G(i+1,j)$; 4. (4)

$F(i,j)\leq F(i+1,j)$ implies $G(i,j)\leq G(i+1,j)$.

It is clear that for all $G\in C$, we have that $G$ is constant on the connected regions on which $F$ is constant. The function $G$ can have arbitrary values on these $d$ connected regions as long as the 4 inequalities above and the linear constraint $\sum_{i,j}G(i,j)=0$ are satisfied. It follows that $\operatorname{gsparse}_{X}(F)=\dim C=d-1$. ∎

12.3. The total variation norm is not tight

Below we denoised the image of the letter C using ROF total variation denoising. The original image has only 2 colors, black and white, and has geometric $X$-sparsity 1. In the denoised images, there are various shades of grey, and the geometric $X$-sparsity is more than 1. So ROF denoising may increase the geometric sparsity, so the norms $\|\cdot\|_{X}$ and $\|\cdot\|_{Y}$ are not tight.

13. Tensor Decompositions

13.1. CP decompositions

One of the motivations for writing this paper is the study of tensor decompositions. Suppose that ${\mathbb{F}}$ is the field ${\mathbb{C}}$ or ${\mathbb{R}}$, and that $V^{(1)},\dots,V^{(d)}$ are finite dimensional ${\mathbb{F}}$-vector spaces. Define

[TABLE]

With a tensor we will mean an element of $V$. Elements of $V$ can be thought of as multi-way arrays of size $n_{1}\times n_{2}\times\cdots\times n_{d}$ where $n_{i}=\dim V^{(i)}$. A simple tensor (also called rank one tensor or simple tensor) is a tensor of the form

[TABLE]

where $v^{(i)}\in V^{(i)}$ for all $i$. Not every tensor is simple, but every tensor can be written as a sum of simple tensors.

Problem 13.1** (Tensor Decomposition).**

Given a tensor $T$, find a decomposition $T=v_{1}+v_{2}+\dots+v_{r}$ where $v_{1},\dots,v_{r}$ are simple tensors and $r$ is minimal.

Hitchcock defined in [31] the rank of the tensor $T$ as the smallest $r$ for which such a decomposition exists and this minimal rank decomposition is called the canonical polyadic decomposition. Problem 13.1 is also known as the PARAFAC ([29]) or CANDECOMP ([8]) model. Finding the rank of a tensor is an NP-hard problem. Over ${\mathbb{Q}}$ this was shown in [30] and in our case, ${\mathbb{F}}={\mathbb{R}}$ or ${\mathbb{F}}={\mathbb{C}}$, this was proved in [32].

13.2. The CoDe model and the nuclear norm

Even in relatively small dimensions, there are examples of tensors for which the rank is unknown. Using the heuristic of convex relaxation, we consider the following problem:

Problem 13.2** (CoDe model).**

Given a tensor $T$, find a decomposition $T=v_{1}+v_{2}+\cdots+v_{r}$ where $v_{1},\dots,v_{r}$ are simple tensors and $\sum_{i=1}^{r}\|v_{i}\|_{2}$ is minimal.

The nuclear norm for tensors was explicitly given [37, 38], but the ideas go back to [25] and [47] :

Definition 13.3**.**

The nuclear norm $\|T\|_{\star}$ of the tensor as the smallest possible value of $\sum_{i=1}^{r}\|v_{i}\|_{2}$ such that $v_{1},\dots,v_{r}$ are simple tensors and $T=\sum_{i=1}^{r}v_{i}$.

A matrix can be viewed as a 2-way tensor and in this case the nuclear norm for tensors coincides with the nuclear norm of the matrix, which is defined as

[TABLE]

where $A^{\star}$ is the complex conjugate transpose of $A$, $\sqrt{A^{\star}A}$ is the unique nonnegative definite Hermitian matrix whose square is $A^{\star}A$ and $\lambda_{1},\lambda_{2},\dots,\lambda_{r}$ are the singular values of $A$. Although finding the nuclear norm of a higher order tensors is also NP-complete (see [23]), it is often easier than determining its rank. In [19] some examples of tensors are given for which the nuclear norm and the optimal decomposition can be computed, but where the rank of the tensors are unknown.

Let $\|\cdot\|_{X}=\|\cdot\|_{\star}$ be the nuclear norm. The dual norm, $\|\cdot\|_{Y}$, is equal to the spectral norm:

Definition 13.4**.**

The spectral norm of a tensor $T$ is defined by

[TABLE]

Finding the spectral norm of a higher-order tensor is also an NP-complete problem (see [32]).

From now on we consider the case ${\mathbb{F}}={\mathbb{C}}$, where $V$ is the tensor product (over ${\mathbb{C}}$) of several finite dimensional Hilbert spaces. We have a positive definite Hermitian inner product $\langle\cdot,\cdot\rangle$ on $V$. A real inner product is given by $\langle\cdot,\cdot\rangle=\Re\langle\cdot,\cdot\rangle_{\mathbb{C}}$.

13.3. Examples of unitangent tensors

The following examples come from [19]:

Example 13.5**.**

The space ${\mathbb{C}}^{p\times q}$ of complex $p\times q$ matrices has the usual basis $e_{i,j}$ where $1\leq i\leq p$ and $1\leq j\leq q$. Define $V={\mathbb{C}}^{p\times q}\otimes{\mathbb{C}}^{q\times r}\otimes{\mathbb{C}}^{r\times p}$ and define the tensor

[TABLE]

This tensor is related to matrix multiplication. It is known that if $\operatorname{rank}(T_{p,q,r})\leq d$, then two $n\times n$ matrices can be multiplied using $O(n^{3\log(d)/\log(pqr)})$ arithmetic operations in ${\mathbb{C}}$. For most $p,q,r$, the rank of $T_{p,q,r}$ is unknown. For example, the best known lower bound for $\operatorname{rank}(T_{3,3,3})$ is 19, and follows from [7]. The best known upper bound is 23 and comes from [35]. It was shown in [19] that $\|T_{p,q,r}\|_{\sigma}=1$ and $\|T_{p,q,r}\|_{\star}=pqr$. Now (8) is a convex decomposition, because

[TABLE]

Also, we have

[TABLE]

so $T_{p,q,r}$ is unitangent. This means that

[TABLE]

is an $X2$-decomposition, a $2Y$-decomposition and an $XY$-decomposition (as well as $2X$, $Y2$ and $YX$) if $0\leq\lambda\leq 1$. If we take $\lambda=\|S\|_{\star}/\|T_{p,q,r}\|_{\star}=\|S\|_{\star}/(pqr)$, then we have $\|S\|_{\star}\leq\|\lambda T_{p,q,r}\|_{\star}$. It follows that we get

[TABLE]

Similarly, we get inequalities

[TABLE]

and

[TABLE]

Example 13.6**.**

Let $\Sigma_{n}$ be the set of permutation of $\{1,2,\dots,n\}$, and for a permutation $\tau$ denote its sign by $\operatorname{sgn}(\tau)$. The determinant tensor is defined by

[TABLE]

It was shown in [19] that $\|\det_{n}\|_{\sigma}=1$, $\|\det_{n}\|_{\star}=n!$ and $\|\det_{n}\|_{2}=\sqrt{n!}$. In particular, $\det_{n}$ is unitangent. Let

[TABLE]

be the permanent tensor. In [19] it was calculated that $\|\operatorname{perm}_{n}\|_{\sigma}=n!/n^{n/2}$, $\|\operatorname{perm}_{n}\|_{\star}=n^{n/2}$ and $\|\operatorname{perm}_{n}\|_{2}=\sqrt{n!}$. The tensor $\operatorname{perm}_{n}$ is also unitangent.

13.4. The diagonal SVD and the slope decomposition

Following [19], we make the following definitions.

Definition 13.7**.**

Suppose that $v_{1},v_{2},\dots,v_{r}$ are simple tensors with $\|v_{i}\|_{2}=1$ for all $i$. For a real number $t\geq 1$ we say that $v_{1},\dots,v_{r}$ are $t$-orthogonal if

[TABLE]

for every simple tensors $w$ with $\|w\|_{2}=1$.

Note that $t$-orthogonality implies orthogonality because we can take $w=v_{j}$ so that

[TABLE]

implies that $v_{j}$ is orthogonal to all $v_{i}$ with $i\neq j$. By Pythagoras’ theorem, orthogonality is equivalent to $1$-orthogonality.

Definition 13.8**.**

The expression

[TABLE]

is called a Diagonal Singular Value Decomposition (DSVD) if $v_{1},\dots,v_{r}$ are $2$-orthogonal simple tensors of length $1$ and $\lambda_{1}\geq\cdots\geq\lambda_{r}>0$.

If (9) is a DSVD, then we have

[TABLE]

Theorem 13.9**.**

Suppose that a tensor $T$ has a DSVD with singular values $\lambda_{1}>\lambda_{2}>\dots>\lambda_{r}>0$ and multiplicities $m_{1},m_{2},\dots,m_{r}$ respectively. Then we can write

[TABLE]

such that

[TABLE]

for all $i$ and

[TABLE]

is a sequence of $2$-orthogonal simple unit tensors. Then the slope decomposition of $T$ is given by

[TABLE]

where

[TABLE]

Proof.

We have

[TABLE]

so we have

[TABLE]

For $i<j$ we have

[TABLE]

So we also have

[TABLE]

This proves that $T=u_{1}+u_{2}+\cdots+u_{s}$ is the slope decomposition.

∎

It was shown in [19] that the tensor $T_{p,q,r}$ has a diagonal singular value decomposition, but $\operatorname{perm}_{n}$ and $\det_{n}$ do not for $n\geq 3$.

13.5. Group algebra tensors

Suppose that $G$ is a finite group of order $n$. Suppose that there are $m_{d}$ irreducible representations of dimension $d$. Then we have $\sum_{d}m_{d}d^{2}=n$. Let $e_{g}$, $g\in G$ be an orthonormal basis of ${\mathbb{C}}^{n}$ and consider the tensor

[TABLE]

which is related to the multiplication in the group algebra. Then $T_{G}$ has singular value $\sqrt{n/d}$ with multiplicity $m_{d}d^{3}$ for all $d$. We have

[TABLE]

In the figure we drew the Pareto sub-frontier of $T_{G}$ for all groups $G$ of order $G$. The blue graph represents the abelian groups ${\mathbb{Z}}/12$ and ${\mathbb{Z}}/6\times{\mathbb{Z}}/2$ with only $1$-dimensional representations, the red graph represents the dihedral group $D_{6}$ and and the semi-direct product ${\mathbb{Z}}/4\ltimes{\mathbb{Z}}/3$ with representations of dimension $1,1,1,1,2,2$, and the yellow graph represents the alternating group ${\bf A}_{4}$ with representations of dimension $1,1,2,3,3$.

13.6. symmetric tensors in ${\mathbb{R}}^{2\times 2\times 2}$

For the remainder of the section, let us consider the tensor product space ${\mathbb{R}}^{2}\otimes{\mathbb{R}}^{2}\otimes{\mathbb{R}}^{2}$. In particular, for $t\in{\mathbb{R}}$, we will study the symmetric tensor

[TABLE]

Proposition 13.10**.**

1. (1)

We have

[TABLE] 2. (2)

We have

[TABLE]

Proof.

(1) By the definition of the spectral norm, we have

[TABLE]

By Banach’s Theorem (see [2, 22]), we may take $v_{1}=v_{2}=v_{3}=v$. If we write $v=xe_{1}+ye_{2}$ with $x^{2}+y^{2}=1$, then we have

[TABLE]

Let $g_{t}(y)=3y+(t-3)y^{3}$ for $y\in[-1,1]$. We get $g_{t}^{\prime}(y)=3+3(t-3)y^{2}$. If $t>2$ then $g_{t}^{\prime}(y)$ has no roots in $[-1,1]$ and

[TABLE]

If $t<2$, then the roots of $g_{t}^{\prime}(y)$ are

[TABLE]

We have

[TABLE]

(2) Note that

[TABLE]

This implies that

[TABLE]

If $t\geq\frac{1}{3}$, then let $s=2-t^{-1}\in[-1,2]$. We get

[TABLE]

So it follows that

[TABLE]

so we must have equality.

For $t=\frac{1}{3}$ we get $\|f_{1/3}\|_{\star}=\frac{8}{3}$. For $t<\frac{1}{3}$ we get

[TABLE]

We have

[TABLE]

so this shows that $\|f_{t}\|_{\star}=3-t$.

∎

Corollary 13.11**.**

For $0\leq t\leq\frac{1}{3}$

[TABLE]

is a $\star\sigma$-decomposition, and for $\frac{1}{3}\leq t\leq\frac{1}{2}$

[TABLE]

is a $\star\sigma$-decomposition. A parametrization of the subpareto curve is given by

[TABLE]

if $0\leq t\leq\frac{1}{3}$ and

[TABLE]

if $\frac{1}{3}\leq t\leq\frac{1}{2}$.

Let us plot the Pareto sub-frontier:

The blue part of the graph is linear, but the red part is non-linear. The graph is not piecewise linear, so $f_{0}$ does not have a slope decomposition. This shows that the nuclear norm and the spectral norm on ${\mathbb{R}}^{2}\otimes{\mathbb{R}}^{2}\otimes{\mathbb{R}}^{2}$ are not tight.

Below we have plotted the singular value region. The singular value $\frac{2}{\sqrt{3}}$ appears with multiplicity $\frac{27}{13}$, the singular value $\frac{1}{4}$ appears with multiplicity $\frac{4}{3}$. The singular values between $\frac{1}{4}$ and $\frac{2}{\sqrt{3}}$ appear with infinitesemal multiplicities. The height of the region is the spectral norm $\|f_{0}\|_{\sigma}=\frac{2}{\sqrt{3}}$, the area of the region is the nuclear norm $\|f_{0}\|_{\star}=3$, and if we integrate $2y^{2}$ over the region we get the square of the euclidean norm which is $\|f_{0}\|_{2}^{2}=3$.