Error Bounds and Singularity Degree in Semidefinite Programming

TL;DR

This paper explores the relationship between error bounds, singularity degree, and convergence speed in semidefinite programming, providing new bounds and insights supported by numerical evidence.

Contribution

It introduces a method to bound the maximum rank of solutions and establishes a link between singularity degree and convergence rate, complementing Sturm's upper bound.

Findings

Large singularity degree can cause slow convergence.

A lower bound on forward error is derived for certain sequences.

Numerical results support the theoretical findings.

Abstract

In semidefinite programming a proposed optimal solution may be quite poor in spite of having sufficiently small residual in the optimality conditions. This issue may be framed in terms of the discrepancy between forward error (the unmeasurable `true error') and backward error (the measurable violation of optimality conditions). In his seminal work, Sturm provided an upper bound on forward error in terms of backward error and singularity degree. In this paper we provide a method to bound the maximum rank over all solutions and use this result to obtain a lower bound on forward error for a class of convergent sequences. This lower bound complements the upper bound of Sturm. The results of Sturm imply that semidefinite programs with slow convergence necessarily have large singularity degree. Here we show that large singularity degree is, in some sense, also a sufficient condition for slow convergence for a family of external-type `central' paths. Our results are supported by numerical observations.

Tables

Table 1. Table 5.1: A record of relevant measures and their bounds for the spectrahedra considered in our analysis. Here r ¯ ¯ 𝑟 \overline{r} , ϵ italic-ϵ \epsilon , and d 𝑑 d denote the upper bound on rank, lower bound on forward error, and lower bound on singularity degree, respectively. The number of different rates of convergence among the eigenvalues of X ​ ( α ) 𝑋 𝛼 {X(\alpha)} that vanish is denoted by N λ subscript 𝑁 𝜆 N_{\lambda} .

$ℱ$	${ϵ}^b\left(ℱ\right)$	$r$	$\overline{r}$	${ϵ}^f\left(ℱ\right)$	$ϵ$	$\mathrm{sd}\left(ℱ\right)$	$d$	$N_{\lambda }$
spec1	$6.62\times {10}^{-11}$	$7$	$7$	$4.36\times {10}^{-11}$	$3.10\times {10}^{-12}$	$1$	$1$	$1$
spec2	$4.44\times {10}^{-13}$	$1$	$1$	$4.93\times {10}^{-2}$	$3.19\times {10}^{-2}$	$4$	$4$	$4$
spec3	$2.47\times {10}^{-13}$	$2$	$2$	$3.39\times {10}^{-2}$	$9.88\times {10}^{-3}$	$5$	$4$	$5$
spec4	$1.88\times {10}^{-11}$	$5$	$5$	$1.35\times {10}^{-5}$	$4.16\times {10}^{-7}$	$\ge 2$	$2$	$2$
spec5	$2.61\times {10}^{-13}$	$2$	$2$	$1.22\times {10}^{-6}$	$6.96\times {10}^{-8}$	$\ge 2$	$2$	$2$

Equations

$${ϵ}^f(X,\mathcal{F}):=\mathrm{dist}(X,\mathcal{F}).$$

$$\begin{array}{rl}{ϵ}^b(X,\mathcal{F})& :=\mathrm{dist}(X,\mathcal{L})+\mathrm{dist}(X,{\mathbb{S}}_{+}^n).\end{array}$$

$$X\approx \left[\begin{array}{ccccc}0.94& 0& 0.028& 0.001& 2.3\times 10^{-6}\\ 0& 0.057& 0& 0& 0\\ 0.028& 0& 0.028& 4.1\times 10^{-5}& 6.5\times 10^{-8}\\ 0.001& 0& 4.1\times 10^{-5}& 4.5\times 10^{-6}& 3.1\times 10^{-9}\\ 2.3\times 10^{-6}& 0& 6.5\times 10^{-8}& 3.1\times 10^{-9}& 0\end{array}\right],\lambda (X)\approx \left(\begin{array}{c}0.94\\ 0.057\\ 1.9\times 10^{-3}\\ 2.4\times 10^{-6}\\ -5.4\times 10^{-12}\end{array}\right),$$

$${ϵ}^f(X,\mathcal{F})=\mathcal{O}\left({ϵ}^b(X,\mathcal{F}{)}^{2^{-\mathrm{sd}(\mathcal{F}))}}\right).$$

$$X,Y\in {\mathbb{S}}_{+}^n,X+Y\in f⟹X,Y\in f.$$

$$WV=0,W+V{V}^T\succ 0,f=V{\mathbb{S}}_{+}^r{V}^T={\mathbb{S}}_{+}^n\cap W^{\perp }.$$

$$f^1⊋\cdots ⊋f^d,f^d=\mathrm{face}(\mathcal{F}).$$

$$f^k=V^k{\mathbb{S}}_{+}^{r_k}{\left(V^k\right)}^T={\mathbb{S}}_{+}^n\cap {\left(W^k\right)}^{\perp },k\in \{1,\dots ,d\}.$$

$$\mathcal{F}=\mathcal{F}(\mathcal{A},b)=\{X\in {\mathbb{S}}_{+}^n:\mathcal{A}(X)=b\}$$

$$\mathcal{F}$$

$$=V\{R\in {\mathbb{S}}_{+}^r:\mathcal{A}(VR{V}^T)=b\}{V}^T$$

$$=V\mathcal{F}\left(\mathcal{A}(V\cdot V^T),b\right)V^T.$$

$$Z^{k+1}=\left[\begin{array}{cc}{Q}_1^{k+1}& Q_2^{k+1}\end{array}\right]\left[\begin{array}{cc}{\Lambda }^{k+1}& 0\\ 0& 0\end{array}\right]\left[\begin{array}{cc}{Q}_1^{k+1}& Q_2^{k+1}\end{array}\right],$$

$${ϵ}^f(X(\alpha ),\mathcal{F})=\{\begin{array}{ll}\mathcal{O}\left({ϵ}^b(X(\alpha ),\mathcal{F})\right)& \text{if }\mathcal{F}=\{0\},\\ \mathcal{O}\left({ϵ}^b(X(\alpha ),\mathcal{F}{)}^{2^{-\mathrm{sd}(\mathcal{F})}}\right)& \text{otherwise.}\end{array}$$

$$\mathrm{face}(Q\mathcal{F}{Q}^T)=\left[\begin{array}{cc}{\mathbb{S}}_{+}^r& 0\\ 0& 0\end{array}\right],$$

$$QX(\alpha )Q^T=\left[\begin{array}{cccc}{X}_0(\alpha )& \ast & \cdots & \ast \\ \ast & X_1(\alpha )& & \\ ⋮& & \ddots & \ast \\ \ast & \ast & \ast & X_{\mathrm{sd}(\mathcal{F})}(\alpha )\end{array}\right],$$

$$X_i(\alpha )\in {\mathbb{S}}^{q_i}\text{ and }\Vert X_i(\alpha )\Vert =\mathcal{O}\left({\alpha }^{\xi (i)}\right),$$

$$\begin{array}{rl}{p}^{\ast }:=\min & ⟨C,X⟩\\ \text{s.t.}& \hat{\mathcal{A}}(X)=\hat{b}\\ & X⪰0,\end{array}$$

$$\mathcal{F}=\{X\in {\mathbb{S}}_{+}^n:\hat{\mathcal{A}}(X)=\hat{b},⟨C,X⟩=p^{\ast }\}.$$

$$\mathcal{A}(X):=\left(\begin{array}{c}\hat{\mathcal{A}}(X)\\ ⟨C,X⟩\end{array}\right),b:=\left(\begin{array}{c}\hat{b}\\ p^{\ast }\end{array}\right).$$

$$\frac{{\lambda }_i(X(\alpha ))}{{\lambda }_{i+1}(X(\alpha ))},i\in \{1,\dots ,n-1\}.$$

$${\mathcal{I}}^1=r+\{1,\dots ,q^1\},{\mathcal{I}}^2=r+q^1+\{1,\dots ,q^2\},$$

$$j\in {\mathcal{I}}^i⟹{\lambda }_j(X(\alpha ))=\{\begin{array}{ll}\Theta (1)& \text{if }i=0,\\ \mathcal{O}\left({\alpha }^{\xi (i)}\right)& \text{otherwise,}\end{array}$$

$$j\in {\mathcal{I}}^0⟹{\lambda }_j(X(\alpha ))=\Theta (1),$$

$$\mathrm{face}(Q\mathcal{F}{Q}^T)=\left[\begin{array}{cc}{\mathbb{S}}_{+}^r& 0\\ 0& 0\end{array}\right]\text{ and }QX(\alpha )Q^T=\left[\begin{array}{cccc}{X}_0(\alpha )& \ast & \cdots & \ast \\ \ast & X_1(\alpha )& & \\ ⋮& & \ddots & \ast \\ \ast & \ast & \ast & X_{\mathrm{sd}(\mathcal{F})}(\alpha )\end{array}\right]\forall \alpha >0,$$

$$X_i(\alpha )\in {\mathbb{S}}^{q_i}\text{ and }\Vert X_i(\alpha )\Vert =\mathcal{O}\left({\alpha }^{\xi (i)}\right).$$

$$S(\alpha ):=\left[\begin{array}{ccc}{X}_i(\alpha )& & \\ & \ddots & \\ & & X_{\mathrm{sd}(\mathcal{F})}(\alpha )\end{array}\right].$$

$$\Vert S(\alpha )\Vert =\mathcal{O}\left(\underset{\ell \in \{i,\dots ,\mathrm{sd}(\mathcal{F})}{\max }\Vert X_{\ell }(\alpha )\Vert \right)=\mathcal{O}\left({\alpha }^{\xi (i)}\right).$$

$${\lambda }_j(X(\alpha ))\le {\lambda }_1(S).$$

$$\underset{k\to \infty }{\mathrm{liminf}}\frac{a_{k+1}}{a_k}\le \underset{k\to \infty }{\mathrm{limsup}}\frac{b_{k+1}}{b_k}.$$

$$\frac{a_{k+1}}{a_k}\ge L_a-\frac{\tau }{3}\text{ and }\frac{b_{k+1}}{b_k}\le L_a-\frac{\tau }{2}$$

Full text

Keywords: Semidefinite programming, SDP, facial reduction, singularity degree, maximizing $\log\det$.

AMS subject classifications: 90C22, 90C25

1 Introduction

It is well known that for certain pathological instances of semidefinite programming, state-of-the-art algorithms, while theoretically guaranteed to converge to a solution, do so very slowly or can fail to converge entirely. This issue is exacerbated in that it is generally undetectable. In this paper we propose a method to detect this type of slow convergence by lower bounding forward error, i.e., distance to the solution set. This bound is obtained by analyzing a class of parametric curves that are proven to converge to a solution of maximum rank and then upper bounding that rank. In the second part of the paper we present a new analysis of the relation between forward error and singularity degree, a measure introduced by Sturm in [25] and shown to be a necessary condition for slow convergence. Our results indicate that large singularity degree is, in some sense, also a sufficient condition for slow convergence for a certain family of central paths.

To be more specific about the type of slow convergence we are concerned with, let ${\mathcal{F}\subset\mathbb{S}^{n}}$ be the solution set of a semidefinite program (SDP). Throughout this paper we refer to $\mathcal{F}$ as a spectrahedron. Here $\mathbb{S}^{n}$ denotes the ambient space of $n\times n$ symmetric matrices. It is always possible to express $\mathcal{F}$ as the intersection of an affine subspace, ${\mathcal{L}}$, and the set of positive semidefinite matrices, $\mathbb{S}_{+}^{n}$. Given a matrix $X\in\mathbb{S}^{n}$, the forward error is defined as,

[TABLE]

We cannot expect to measure forward error accurately without substantial knowledge of $\mathcal{F}$. For this reason forward error is generally unknown. What is readily available to users is backward error,

[TABLE]

In backward error we recognize that $\mathcal{F}$ is the intersection of two sets with easily computable forward errors. Backward error serves as a proxy for the unknown forward error. The type of slow convergence we are concerned with is when backward error is sufficiently small but forward error is much larger. The problem with this scenario is not just the poor quality of the proposed solution. More than this, it is the lack of awareness of a poor solution.

To demonstrate the discrepancy between forward error and backward error, we consider an SDP, with $n=5$, from the family introduced in [26]. The output of cvx using the solver SDPT3 is,

[TABLE]

where $\lambda(X)$ is the vector of eigenvalues of $X$. Similar results were obtained with the solvers SeDuMi and MOSEK. The backward error for $X$ is quite small at ${5.46\times 10^{-12}}$ and cvx output states that the problem is “solved”. All indicators point to a ‘good’ solution. However, the solution set of the SDP is a singleton consisting of the matrix with $1$ in the upper left entry and zeros everywhere else. Given this information, $X$ does not look like a very good solution. Indeed, forward error is ${9.15\times 10^{-2}}$, unacceptably large. Moreover, the eigenvalues of $X$ hardly indicate that the solution is a rank one matrix. In the numerical case studies of Section 5, we show that our lower bound on forward error is significantly greater than backward error. Therefore, it serves as an alarm that the proposed solution is not as accurate as it appears to be.

In [25], Sturm defined singularity degree as the fewest number of iterations required in the facial reduction algorithm, a regularization scheme for conic optimization introduced in [3, 2, 4]. The singularity degree of a spectrahedron $\mathcal{F}$, denoted $\operatorname{sd}(\mathcal{F})$, is an integer between [math] and $n-1$. Sturm showed that forward error is bounded in terms of backward error and singularity degree,

[TABLE]

In particular, this bound implies that large singularity degree is a necessary condition for large forward error. Equivalently, small singularity degree implies small forward error. It is exactly this relation that has motivated our study of singularity degree.

The challenge with singularity degree is that, like forward error, it is unknown in most cases. In [5] it is shown that the facial reduction algorithm is stable when singularity degree is [math] or $1$, but the authors were not able to show that stability holds for larger singularity degree. Moreover, the empirical evidence we have obtained indicates a lack of stability of the algorithm when singularity degree is greater than $1$. For this reason, we view singularity degree as intractable for general instances of SDP. Here we obtain a lower bound on singularity degree as a consequence of the upper bound on maximum rank.

Our final contribution in this work is showing that singularity degree is also a sufficient measure, in some sense, for large forward error. We prove that for a class of central paths the eigenvalues that vanish, do so at a ‘fast rate’ if, and only if, singularity degree is at most $1$. We also prove that among the elements of the dual path that converge to [math], there are at least $\operatorname{sd}(\mathcal{F})$ different rates of convergence.

The paper is organized as follows. In Section 2 we introduce our notation and basic concepts pertaining to facial reduction and singularity degree. The bounds on maximum rank, forward error, and singularity degree are presented in Section 3 and in Section 4 we present results that support the notion that singularity degree is a measure of hardness. The paper is concluded with numerical observations in Section 5.

2 Notation and Background

Throughout this paper the ambient space is the Euclidean space of $n\times n$ real symmetric matrices, denoted $\mathbb{S}^{n}$, with the standard trace inner product, $\langle X,Y\rangle:=\operatorname{{trace}}(XY)$, and the induced Frobenius norm, $\lVert X\rVert_{F}:=\sqrt{\langle X,X\rangle}$.

The eigenvalues of any $X\in\mathbb{S}^{n}$ are real and ordered so as to satisfy, $\lambda_{1}(X)\geq\cdots\geq\lambda_{n}(X)$, and $\lambda(X)\in\mathbb{R}^{n}$ is the vector consisting of all the eigenvalues. In terms of this notation we have $\lVert X\rVert_{F}=\lVert\lambda(X)\rVert_{2}$, where $\lVert\cdot\rVert_{2}$ is the Euclidian norm when the argument is a vector in $\mathbb{R}^{n}$. When the argument to $\lVert\cdot\rVert_{2}$ is a symmetric matrix then we mean the operator 2-norm, defined as $\lVert X\rVert_{2}:=\max_{i}\lvert\lambda_{i}(X)\rvert$. In some of our discussion we use the notation ${\lambda_{\max}(X)=\lambda_{1}(X)}$ and ${\lambda_{\min}(X)=\lambda_{n}(X)}$ if we are not concerned with the dimensions of the matrix, or wish to stress the minimality and maximality of the values.

The set of positive semidefinite matrices, $\mathbb{S}_{+}^{n}$, is a closed convex cone in $\mathbb{S}^{n}$, with interior consisting of the positive definite matrices, $\mathbb{S}_{++}^{n}$. The cone $\mathbb{S}_{+}^{n}$ induces the Löwner partial order on $\mathbb{S}^{n}$. That is, for $X,Y\in\mathbb{S}^{n}$ we write $X\succeq Y$ when $X-Y\in\mathbb{S}_{+}^{n}$ and similarly $X\succ Y$ when ${X-Y\in\mathbb{S}_{++}^{n}}$.

2.1 Facial Reduction for SDPs

To introduce facial reduction, we begin with a brief discussion of the faces of $\mathbb{S}_{+}^{n}$. For further reading and proofs of some of our claims, we suggest [30, 26, 23]. A face of $\mathbb{S}_{+}^{n}$, say $f$, is a convex subset of $\mathbb{S}_{+}^{n}$ such that,

[TABLE]

For a face $f$ there exists ${r\in\{0,\dotso,n\}}$, ${W\in\mathbb{S}_{+}^{n}}$, and ${V\in\mathbb{R}^{n\times r}}$ such that,

[TABLE]

When the matrix $W$ in (2.1) is not the [math] matrix, it is referred to as an exposing vector for $f$.

An important notion regarding SDPs is that of minimal face. The minimal face of $\mathbb{S}_{+}^{n}$ containing a convex set $C$, denoted $\operatorname{face}(C)$, is the intersection of all faces of $\mathbb{S}_{+}^{n}$ that contain $C$. If the minimal face for an SDP is known, then the SDP may be transformed into an equivalent SDP for which the Slater condition – strict feasibility with respect to the positive semidefinite constraint – holds. See the survey [9] or [19, 20, 21, 27] for further reading on regularization of SDPs via facial reduction.

For SDPs with special structure, the minimal face may be obtained through theoretical analysis. Alternatively, the minimal face may be obtained using the facial reduction algorithm, introduced in [3, 2, 4]. This algorithm generates a sequence of faces $f^{1},\cdots,f^{d}$ satisfying,

[TABLE]

Equivalently, using the two characterizations of faces in (2.1), the algorithm generates a sequence of matrices ${W^{1},\dotso,W^{d}}$, a sequence of decreasing positive integers ${r_{1},\dotso,r_{d}}$, and a sequence of matrices ${V^{k}\in\mathbb{R}^{n\times r_{k}}}$ with ${k\in\{1,\dotso,d\}}$ such that,

[TABLE]

The facial reduction algorithm depends on the algebraic representation of the solution set of an SDP. As stated earlier, such a set is the intersection of an affine subspace and $\mathbb{S}_{+}^{n}$. We assume that the affine subspace is defined in terms of a linear map ${\mathcal{A}}:\mathbb{S}^{n}\to{\mathbb{R}^{m}\,}$ and a vector $b\in{\mathbb{R}^{m}\,}$ so that,

[TABLE]

The notation $\mathcal{F}({\mathcal{A}},b)$ stresses the dependence on the algebraic representation of the the affine subspace. The facial reduction algorithm relies on the following theorem of the alternative.

Fact 2.1**.**

Let $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ be defined as in (2.2) and nonempty. Then exactly one of the following holds:

1. $(i)$

$\mathcal{F}\cap\mathbb{S}_{++}^{n}\neq\emptyset$, 2. $(ii)$

there exists nonzero $W={\mathcal{A}}^{*}(y)$ with $y^{T}b=0$.

A proof of this result may be found in [9], for instance.

Now let us briefly describe how this result may be used for facial reduction. Recall that the purpose of the facial reduction algorithm is to create an equivalent representation of $\mathcal{F}$ so that the Slater condition holds. Now if Fact 2.1 i holds, then $\mathcal{F}$ has a Slater point and we are done. On the other hand if Fact 2.1 ii holds, then it can be verified that the matrix $W$ is an exposing vector for a face containing $\operatorname{face}(\mathcal{F})$. Indeed, the definition of $W$ implies that $\langle X,W\rangle=0$ for all ${X\in\mathcal{F}}$. Letting $r$ be the nullity of $W$ and choosing $V$ so as to satisfy the properties in (2.1), we see that for all $X\in\mathcal{F}$ we have $X\in V\mathbb{S}_{+}^{r}V^{T}$. It follows that,

[TABLE]

In the next iteration of the algorithm we apply Fact 2.1 to the spectrahedron $\mathcal{F}\left({\mathcal{A}}(V\cdot V^{T}),b\right)$. We continue in this way until eventually Fact 2.1 i – the Slater condition – holds. We have included Algorithm 2.1 as a more rigorous description of facial reduction.

At each iteration, the order of the matrices is reduced by at least one, implying that the algorithm terminates in at most $n$ iterations. In fact, the upper bound is actually $n-1$ since in the case $\mathcal{F}=\{0\}$ it can be shown that the algorithm terminates in exactly $1$ iteration.

2.2 Singularity Degree and the Bounds of Sturm

The number of iterations required by the facial reduction algorithm of the previous section is dependent on the choice of exposing vector, $Z^{k+1}$, obtained at each iteration. When the exposing vector is chosen to have maximum rank, the number of iterations generated by Algorithm 2.1 is defined as the singularity degree, and denoted $\operatorname{sd}(\mathcal{F})$. It can be shown that singularity degree is the least number of iterations required by the algorithm.

Two special cases deserve mention. The first case is that of $\operatorname{sd}(\mathcal{F})=0$. This case occurs if, and only if, the solution set satisfies the Slater condition. The second special case is when ${\mathcal{F}=\{0\}}$. Here our definition of singularity degree does not coincide with that of Sturm. By our definition we have $\operatorname{sd}(\mathcal{F})=1$, since exactly one iteration of the algorithm is required to obtain the Slater condition. On the other hand, Sturm defines $\operatorname{sd}(\mathcal{F})=0$ for this case, based on his error bounds. That is, the worst case error bounds when $\mathcal{F}=\{0\}$ are the same as when $\mathcal{F}$ satisfies the Slater condition.

We now state the error bounds of Sturm. For a proof, see Theorem 3.3 of [25].

Fact 2.2**.**

Let $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ be a nonempty spectrahedron and let $\{X(\alpha):\alpha>0\}$ be a sequence where $\lVert X(\alpha)\rVert$ is bounded. Then,

[TABLE]

In proving Fact 2.2, Sturm actually obtained the following more precise statement about the way in which ${X(\alpha)}$ approaches $\mathcal{F}$.

Fact 2.3**.**

Let $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ be a nonempty spectrahedron with ${\operatorname{sd}(\mathcal{F})\geq 1}$ where ${\mathcal{F}\neq\{0\}}$ and let ${\{{X(\alpha)}:\alpha>0\}}$ be a sequence where $\epsilon^{b}({X(\alpha)},\mathcal{F})={\mathcal{O}}(\alpha)$. For ${i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}}$, let $Z^{i}$ be a maximum rank exposing vector obtained as in Algorithm 2.1 and let ${q_{i}:=\operatorname{{rank}}(Z^{i})}$. Let $\bar{\alpha}>0$ be fixed. Then there exists an orthogonal matrix $Q$ such that,

[TABLE]

and,

[TABLE]

where $X_{0}(\alpha)\in\mathbb{S}^{r}$ and for all $i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}$ and $\alpha\in(0,\bar{\alpha})$ it holds that,

[TABLE]

where $\xi(i):=2^{-(\operatorname{sd}(\mathcal{F})-i)}$.

This result shows that under the correct orthogonal transformation, the diagonal blocks of ${X(\alpha)}$ that converge to [math] may do so at different rates.

3 Bounds on Maximum Rank, Forward Error, and Singularity Degree

We consider SDPs in the form,

[TABLE]

where $\widehat{\mathcal{A}}:\mathbb{S}^{n}\to\mathbb{R}^{\widehat{m}}$ is a linear map, $\widehat{b}\in\mathbb{R}^{\widehat{m}}$, and $C\in\mathbb{S}^{n}$. The solution set is,

[TABLE]

When (3.1) is a feasibility problem then $C=0$ and $p^{*}=0$. In this case we let $m:=\widehat{m}$, ${\mathcal{A}}:=\widehat{\mathcal{A}}$, and $b:=\widehat{b}$ so that ${\mathcal{F}=\mathcal{F}({\mathcal{A}},b)=\mathcal{F}(\widehat{\mathcal{A}},\widehat{b})}$. On the other hand, when the objective of (3.1) is non-trivial, i.e., $C\neq 0$, then it may not be the case that ${\mathcal{F}=\mathcal{F}(\widehat{\mathcal{A}},\widehat{b})}$. Thus we define $m:=\widehat{m}+1$ and define ${\mathcal{A}}$ and $b$ as,

[TABLE]

Then $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ is a spectrahedron in the notation we have already developed.

Assumption 3.1**.**

Let $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ be the spectrahedron consisting of the solutions to the SDP in (3.1). We assume that $\mathcal{F}$ is nonempty, ${\operatorname{sd}(\mathcal{F})\geq 1}$, and ${\mathcal{F}\neq\{0\}}$.

The assumption that $\mathcal{F}$ is nonempty is for the purpose of defining singularity degree. The other two assumptions ensure that there is a possibility of discrepancy between forward error and backward error.

Our analysis in this section is based on path-following algorithms for SDP. The foundation of such algorithms is the central path, a smooth parametric curve, say ${\{X(\alpha):\alpha>0\}}$, that is known to converge to a solution of the SDP, granted that a solution exists. Specifically we mean that ${{X(\alpha)}\to\bar{X}}$ as ${\alpha\searrow 0}$ and ${\bar{X}\in\mathcal{F}}$. Then a path-following algorithm produces a sequence of positive numbers $\{\alpha_{k}\}$ and matrices $\{X^{k}\}$ such that $\alpha_{k}$ is successively closer to [math] and $X^{k}$ is a successively better approximation of $X(\alpha_{k})$. In other words, the iterates approach the solution set along a trajectory that approximates the central path.

To obtain a method for bounding maximum rank and forward error, we study central paths that satisfy the following assumptions.

Assumption 3.2**.**

Let $\{X(\alpha):\alpha>0\}$ be a central path and let $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ be the spectrahedron consisting of the solutions to the SDP in (3.1). We assume that,

1. $(i)$

there exists $\bar{X}\in\operatorname{{relint}}(\mathcal{F})$ such that $\lim_{\alpha\searrow 0}{X(\alpha)}=\bar{X},$ 2. $(ii)$

${X(\alpha)}\succ 0$* for all $\alpha>0$.*

Many of the well-known algorithms for SDP are based on central paths that satisfy this assumption.

3.1 A Bound on Maximum Rank

We provide two ways to bound maximum rank. The first method is based on tracking the ratios,

[TABLE]

The ratios that blow up indicate one of two scenarios. The first scenario is that both eigenvalues converge to [math], but $\lambda_{i+1}({X(\alpha)})$ does so much more quickly. The second is that $\lambda_{i}({X(\alpha)})$ converges to a positive value and $\lambda_{i+1}({X(\alpha)})$ vanishes. This only happens when $i$ corresponds to the rank of the limit point $\bar{X}$. Thus the smallest index $i$ for which the ratio blows up corresponds exactly to the maximum rank. We state this observation formally in the following.

Proposition 3.3**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 3.1. Let $i$ be the smallest index such that the ratio in (3.3) blows up. Then $i$ is the maximum rank over $\mathcal{F}$.

In practice, a ratio of the form in (3.3) may blow up slowly, so that it may appear to be bounded. In such cases an approximation of the maximum rank may be obtained by the smallest index $i$ for which it is clear that the ratio blows up.

In addition to differentiating between eigenvalues that vanish and those that do not, the ratios or (3.3) also indicate the number of different rates of convergence among the eigenvalues that do converge to [math]. We explore this further in Section 4.2.

The second method to bound maximum rank is more involved, but provides us with additional information on singularity degree. Moreover, this method appears to be more reliable in our experiments. The approach is to analyze the $Q$-convergence rates of the eigenvalues of ${X(\alpha)}$. We begin by translating Fact 2.3 into a statement about the eigenvalues of ${X(\alpha)}$.

Lemma 3.4**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ that satisfies Assumption 3.1. Assume further that ${\epsilon^{b}({X(\alpha)},\mathcal{F})={\mathcal{O}}(\alpha)}$. For ${i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}}$, let $Z^{i}$ be a maximum rank exposing vector obtained as in Algorithm 2.1 and let ${q_{i}:=\operatorname{{rank}}(Z^{i})}$. Let $r$ denote the maximum rank over $\mathcal{F}$. Let ${\mathcal{I}}^{0},{\mathcal{I}}^{1},\dotso,{\mathcal{I}}^{\operatorname{sd}(\mathcal{F})}$ form a partition of $\{1,\dotso,n\}$ such that ${\mathcal{I}}^{0}=\{1,\dotso,r\}$ and,

[TABLE]

and so on. Then, for $j\in\{1,\dotso,n\}$ it holds that for sufficiently small $\alpha>0$,

[TABLE]

where $\xi(i):=2^{-(\operatorname{sd}(\mathcal{F})-i)}$.

Proof.

By assumption, ${X(\alpha)}\to\bar{X}\in\operatorname{{relint}}(\mathcal{F})$ and $\operatorname{{rank}}(\bar{X})=r$. Therefore for sufficiently small $\alpha>0$, the $r$ largest eigenvalues of ${X(\alpha)}$ are converging to positive numbers. It follows that for sufficiently small $\alpha>0$,

[TABLE]

proving one part of the desired result.

Next, by Fact 2.3 there exists an orthogonal $Q$ such that,

[TABLE]

where $X_{0}(\alpha)\in\mathbb{S}^{r}$ and for all $i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}$ it holds that,

[TABLE]

Now let $i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}$ and let $j\in{\mathcal{I}}^{i}$. Consider the principal submatrix of $Q{X(\alpha)}Q^{T}$,

[TABLE]

By Assumption 3.2, ${X(\alpha)}\succ 0$ implying that $S(\alpha)\succ 0$. Thus by (3.5) we have,

[TABLE]

It follows that $\lambda_{1}(S)={\mathcal{O}}\left(\alpha^{\xi(i)}\right)$. Moreover, the interlacing eigenvalue theorem implies that,

[TABLE]

Combining these inequalities yields the desired result, $\lambda_{j}({X(\alpha)})={\mathcal{O}}\left(\alpha^{\xi(i)}\right)$. ∎

Now that we have bounds on those eigenvalues of ${X(\alpha)}$ that converge to [math] we analyze their $Q$-convergence rates. First a lemma.

Lemma 3.5**.**

Let $\{a_{k}\}_{k\in{\mathbb{N}}}$ and $\{b_{k}\}_{k\in{\mathbb{N}}}$ be sequences of positive reals such that ${a_{k}\to 0}$ and ${b_{k}\to 0}$. If ${a_{k}\leq b_{k}}$ for all ${k\in{\mathbb{N}}}$ then,

[TABLE]

Proof.

Let $L_{a}$ and $L_{b}$ denote the limit inferior and limit superior of (3.7), respectively. For simplicity we assume that $L_{a}$ and $L_{b}$ are finite, but the arguments extend to the general case trivially. Suppose for the sake of contradiction that there exists $\tau>0$ such that ${L_{a}-\tau\geq L_{b}}$. Then there exists ${\bar{k}\in{\mathbb{N}}}$ such that for all ${k\geq\bar{k}}$,

[TABLE]

Rearranging the first equation in (3.8) gives us,

[TABLE]

Replacing $k$ with $k-1$ we get that,

[TABLE]

Combining (3.9) with (3.10) yields,

[TABLE]

Continuing in this fashion we get,

[TABLE]

Through an analogous approach applied to the second equation of (3.8) we get,

[TABLE]

Combining the hypothesis that $b_{k}$ dominates $a_{k}$ with (3.11) and (3.12) we get,

[TABLE]

Observe that $L_{b}\geq 0$ since $b_{k}\geq 0$ for every $k\in{\mathbb{N}}$. Therefore, ${L_{a}-\tau\geq 0}$ and we have ${L_{a}-\frac{\tau}{3}>L_{a}-\frac{\tau}{2}>0}$. It follows that for sufficiently large $k$, the inequality in (3.13) is violated, giving us the desired contradiction. ∎

Our main result on $Q$-convergence rates of eigenvalues considers sequences of the form $\{\sigma_{k}\}$ for some $\sigma\in(0,1)$ where the $k$th term, $\sigma_{k}$, is the $k$th power of $\sigma$, i.e. $\sigma^{k}$.

Theorem 3.6**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ that satisfies Assumption 3.1. Assume further that ${\epsilon^{b}({X(\alpha)},\mathcal{F})={\mathcal{O}}(\alpha)}$. Let ${{\mathcal{I}}^{0},{\mathcal{I}}^{1},\dotso,{\mathcal{I}}^{\operatorname{sd}(\mathcal{F})}}$ be a partition of $\{1,\dotso,n\}$ as in Lemma 3.4. For ${\sigma\in(0,1)}$ the following hold.

1. $(i)$

If ${j\in{\mathcal{I}}^{0}}$ then,

[TABLE] 2. $(ii)$

If ${j\in{\mathcal{I}}^{i}}$ with $i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}$ then,

[TABLE]

where $\xi(i)=2^{-(\operatorname{sd}(\mathcal{F})-i)}$.

Proof.

By Assumption 3.2 and the definition of ${\mathcal{I}}^{0}$ we have that $\lambda_{j}(X(\sigma^{k}))$ converges, in $k$, to a positive number whenever $j\in{\mathcal{I}}^{0}$. We have proved i.

Now let $j\in{\mathcal{I}}^{i}$ with $i\in\{1,\dotso,\operatorname{sd}(\mathcal{F})\}$. By Lemma 3.4 we have,

[TABLE]

Thus there exists $M>0$ such that $\lambda_{j}(X(\sigma^{k}))\leq M\sigma^{k\xi(i)}$. Now the sequences $\{\lambda_{j}(X(\sigma^{k}))\}_{k\in{\mathbb{N}}}$ and $\{M\sigma^{k\xi(i)}\}_{k\in{\mathbb{N}}}$ satisfy the assumptions of Lemma 3.5. Therefore,

[TABLE]

Lastly $\sigma^{\xi(d)}<1$ holds since, $\sigma\in(0,1)$ and $\xi(i)>0$. ∎

The following corollary emphasizes how Theorem 3.6 may be used to upper bound the rank of $\bar{X}$.

Corollary 3.7**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ that satisfies Assumption 3.1. Assume further that ${\epsilon^{b}({X(\alpha)},\mathcal{F})={\mathcal{O}}(\alpha)}$. Let $r$ denote the maximum rank over $\mathcal{F}$ and let ${\sigma\in(0,1)}$. Then,

[TABLE]

The number $\sigma^{2^{-(\operatorname{sd}(\mathcal{F})-1)}}$ serves as a threshold so that limit inferiors of the eigenvalue ratios lie below this number if, and only if, those eigenvalues converge to [math]. For large singularity degree, it may be difficult to distinguish $\sigma^{2^{-(\operatorname{sd}(\mathcal{F})-1)}}$ from $1$, numerically. However if we can identify another number, say $\tau\in(0,1)$, that is numerically distinguishable from $1$ and there exists a positive integer $\overline{r}$ such that,

[TABLE]

then $\overline{r}$ is an upper bound on the maximum rank, $r$, over $\mathcal{F}$. We state this result formally in the following.

Corollary 3.8**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ that satisfies Assumption 3.1. Assume further that ${\epsilon^{b}({X(\alpha)},\mathcal{F})={\mathcal{O}}(\alpha)}$. Let $r$ denote the maximum rank over $\mathcal{F}$ and let ${\sigma\in(0,1)}$. Suppose there exists ${\tau\in(0,1)}$ and ${\overline{r}\in\{1,\dotso,n\}}$ such that

[TABLE]

Then $\overline{r}\geq r$.

Remark 3.9**.**

In the results of this section we require knowledge of the magnitude of backward error for $\mathcal{F}$. When the SDP in (3.1) has a non-trivial objective, we cannot expect to know the optimal value $p^{*}$. Consequently, backward error is intractable. However, primal dual algorithms measure the duality gap which is an overestimate for $(\langle C,X\rangle-p^{*})$. Thus,

[TABLE]

In particular, for the central path $\{{X(\alpha)}:\alpha>0\}$ it holds that,

[TABLE]

the assumed bound in the above results.

3.2 Bounds on Forward Error and Singularity Degree

Any bound on maximum rank, such as the one from the previous section, may be used to provide lower bounds on forward error and singularity degree.

Theorem 3.10**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ that satisfies Assumption 3.1. If $\overline{r}$ is an upper bound for the maximum rank over $\mathcal{F}$ then for all $\alpha>0$ it holds that,

[TABLE]

Proof.

Let $\overline{r}$ be as in the hypothesis and let $\alpha>0$. Since $\mathcal{F}$ is a closed convex set, there exists $X\in\mathcal{F}$ such that,

[TABLE]

Then observing that $\lVert S\rVert_{F}=\lVert\lambda(S)\rVert_{2}$ for any $S\in\mathbb{S}^{n}$ we have,

[TABLE]

Applying the classical bound $\langle X,Y\rangle\leq\lambda(X)^{T}\lambda(Y)$, e.g. [10, 14], we get,

[TABLE]

Taking the square root of both sides yields the desired result. ∎

Theorem 3.11**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be a central path satisfying Assumption 3.2 for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ that satisfies Assumption 3.1. Assume further that ${\epsilon^{b}({X(\alpha)},\mathcal{F})={\mathcal{O}}(\alpha)}$. Let $\overline{r}$ be an upper bound on the maximum rank over $\mathcal{F}$ and let ${\sigma\in(0,1)}$. Suppose $\underline{d}$ is the smallest positive integer such that,

[TABLE]

Then $\underline{d}\leq\operatorname{sd}(\mathcal{F})$.

Proof.

Let $r$ denote the maximum rank over $\mathcal{F}$. Suppose $\overline{r}>r$. Then by Corollary 3.7 and (3.15) we have,

[TABLE]

It follows that $\underline{d}\leq\operatorname{sd}(\mathcal{F})$. Now suppose that $\overline{r}=r$. Then by Corollary 3.7 we have,

[TABLE]

Out of all positive integers that could replace $\operatorname{sd}(\mathcal{F})$ in (3.16), we chose $\underline{d}$ to be the smallest. Hence ${\underline{d}\leq\operatorname{sd}(\mathcal{F})}$, as desired. ∎

4 Singularity Degree as a Measure of Hardness

It is certainly possible to construct a parametric curve $\{{X(\alpha)}:\alpha>0\}$ with the properties of Assumption 3.2, for which singularity degree is large, but forward error is small. For instance, the path defined as ${{X(\alpha)}:=\bar{X}+\alpha I}$, where $\bar{X}\in\operatorname{{relint}}(\mathcal{F})$, exhibits fast convergence and low forward error irrespective of the singularity degree. This demonstrates that singularity degree is not a sufficient condition for slow convergence for all parametric curves. However, for many of the central paths constructed by state of the algorithms, empirical evidence indicates otherwise. In this section we present several results that give credence to the notion that singularity degree is a measure of hardness for a family of central paths. In Section 4.1 we introduce the family of central paths and in Section 4.2 we present the main results of the section.

4.1 Analaysis of a Family of Central Paths

The classical interior point method for SDPs is based on a central path that is constructed by assuming that both the primal and the dual satisfy the Slater condition. As this assumption is quite restrictive, infeasible central paths assuming weaker conditions have been subsequently proposed. Among these are [18, 7, 22, 6, 16].

In [22], Potra and Sheng proposed a family of infeasible central paths that are based on perturbing the feasible region so as to satisfy the Slater condition, and then decreasing the perturbation. Our analysis requires a complete knowledge of the algebraic representation of the spectrahedron. Thus we assume that $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ for a known linear map ${\mathcal{A}}$ and vector $b$. In terms of the SDP of (3.1), this equates to feasibility problems, i.e., ${C=0}$ and ${\mathcal{A}}:=\widehat{\mathcal{A}}$, $b:=\widehat{b}$. From the family of paths proposed by Potra and Sheng we choose the path $\{{X(\alpha)}:\alpha>0\}$ defined by,

[TABLE]

where $B\succ 0$ is fixed. The matrix ${X(\alpha)}$ exists for each $\alpha>0$ if, and only if, $\mathcal{F}(\alpha)$ is nonempty and bounded. Hence the following assumption.

Assumption 4.1**.**

Let $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ be a spectrahedron defined in terms of a map ${\mathcal{A}}:\mathbb{S}^{n}\to{\mathbb{R}^{m}\,}$ and a vector $b\in{\mathbb{R}^{m}\,}$. We assume that,

1. $(i)$

$\mathcal{F}$* is nonempty, bounded, ${\operatorname{sd}(\mathcal{F})\geq 1}$, and ${\mathcal{F}\neq\{0\}}$,* 2. $(ii)$

${\mathcal{A}}$* is surjective.*

Assumption 4.1 differs from Assumption 3.1 in the additional requirements that $\mathcal{F}$ is bounded and that ${\mathcal{A}}$ is surjective. The need for a bounded $\mathcal{F}$ has already been discussed and the restriction on ${\mathcal{A}}$ ensures a unique $y\in{\mathbb{R}^{m}\,}$ for every $Z\in\operatorname{range}({\mathcal{A}}^{*})$, a property that will prove convenient in the subsequent discussion.

It is easy to see that if $\mathcal{F}\neq\emptyset$ then $\mathcal{F}(\alpha)$ has a Slater point for every $\alpha>0$. For instance, the set ${\mathcal{F}+\alpha B}$ has positive definite elements and is contained in $\mathcal{F}(\alpha)$. Since ${X(\alpha)}$ is chosen to be the determinant maximizer over $\mathcal{F}(\alpha)$ it follows that ${{X(\alpha)}\succ 0}$ and ${X(\alpha)}\in\operatorname{{relint}}(\mathcal{F})$ for each $\alpha>0$. We have thus shown that this central path satisfies Assumption 3.2 ii. In the remainder of this section we show that it also possesses the other properties of Assumption 3.2. Namely, that $\{{X(\alpha)}:\alpha>0\}$ is smooth and converges to a matrix in $\operatorname{{relint}}(\mathcal{F})$ as ${\alpha\searrow 0}$.

The optimality conditions for (4.1) yield the primal-dual central path,

[TABLE]

This primal-dual central path is smooth around every $\alpha>0$ and admits the following convergence result.

Theorem 4.2**.**

Let ${\{({X(\alpha)},{y(\alpha)},{Z(\alpha)}):\alpha>0\}}$ be the primal-dual central path of (4.2) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. Then,

[TABLE]

with

[TABLE]

Proof.

While we are not aware of this exact result in the literature, it may be deduced from two known results. Firstly, in [11] the authors show that every limit point of the primal-dual central path exhibits the properties of (4.3). Then a lemma of Milnor [17] may be used as in [12, 13] to show that the set of limit points is a singleton. ∎

An immediate consequence of Theorem 4.2 is a statement about the convergence rates of eigenvalues of ${X(\alpha)}$ and ${Z(\alpha)}$.

Corollary 4.3**.**

Let ${\{({X(\alpha)},{y(\alpha)},{Z(\alpha)}):\alpha>0\}}$ be the primal-dual central path of (4.2) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. Let ${(\bar{X},\bar{y},\bar{Z})}$ be the limit point of the primal-dual central path and let $r$ and $q$ denote the rank of $\bar{X}$ and $\bar{Z}$, respectively. Let ${\bar{\alpha}>0}$. Then for every $\alpha\in(0,\bar{\alpha})$ we have,

[TABLE]

and

[TABLE]

Proof.

The convergence result of Theorem 4.2 implies that the $r$ largest eigenvalues of ${X(\alpha)}$ and the $q$ largest eigenvalues of ${Z(\alpha)}$ converge to positive values, hence are $\Theta(1)$. Moreover, the relation ${Z(\alpha)}=\alpha{X(\alpha)}^{-1}$ gives that the $q$ smallest eigenvalues of ${X(\alpha)}$ and the $r$ smallest eigenvalues of ${Z(\alpha)}$ converge to [math] at a rate that is $\Theta(\alpha)$.

Now let $i\in\{r+1,\dotso,n-q\}$. The lower bound $\Omega(\alpha)$ holds since,

[TABLE]

Now suppose, for the sake of contradiction, that ${\lambda_{i}({X(\alpha)})={\mathcal{O}}(\alpha)}$. Combining with the lower bound, $\Omega(\alpha)$, we conclude that ${\lambda_{i}({X(\alpha)})=\Theta(\alpha)}$. Once again using the relation ${Z(\alpha)}=\alpha{X(\alpha)}^{-1}$ implies that there are $q+1$ eigenvalues of ${Z(\alpha)}$ that are bounded away from [math], contradicting the assumption that $\operatorname{{rank}}(\bar{Z})=q$ and the statement of Theorem 4.2:

[TABLE]

The remaining bounds on ${Z(\alpha)}$ are obtained in similar fashion. ∎

4.2 On Singularity Degree and Slow Convergence

An immediate implication of Corollary 4.3 is that fast convergence of eigenvalues does not occur when singularity degree is greater than $1$.

Theorem 4.4**.**

Let ${\{{X(\alpha)}:\alpha>0\}}$ be the central path of (4.1) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. If $\operatorname{sd}(\mathcal{F})>1$, there exists an eigenvalue of ${X(\alpha)}$ that converges to [math] at a rate that is not ${\mathcal{O}}(\alpha)$.

For the second theorem of this section we need the following technical results.

Lemma 4.5**.**

Let ${\{({X(\alpha)},{y(\alpha)},{Z(\alpha)}):\alpha>0\}}$ be the primal-dual central path of (4.2) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. Suppose that $b\neq 0$ and let ${v^{1},\dotso,v^{m-1}\in{\mathbb{R}^{m}\,}}$ form a basis for $b^{\perp}$. Let $\bar{\alpha}>0$ and for all $\alpha\in(0,\bar{\alpha})$ let ${\beta(\alpha)}$ and ${\nu_{1}(\alpha),\dotso,\nu_{m-1}(\alpha)}$ be real coefficients such that,

[TABLE]

Then for all $\alpha\in(0,\bar{\alpha})$,

[TABLE]

Proof.

By definition of $b(\alpha)$ in (4.1) and by (4.2) we have,

[TABLE]

Now $\langle B,{\mathcal{A}}^{*}(y(\alpha))\rangle$ is bounded below and above on $\alpha\in(0,\bar{\alpha})$, implying that ${\lvert y(\alpha)^{T}b\rvert=\Theta(\alpha)}$. On the other hand, ${\{v^{1},v^{2},\dotso,v^{m-1}\}\in b^{\perp}}$ by construction. Therefore ${y(\alpha)^{T}b=\beta(\alpha)\lVert b\rVert^{2}}$, yielding,

[TABLE]

as desired. ∎

Lemma 4.6**.**

Let ${\{({X(\alpha)},{y(\alpha)},{Z(\alpha)}):\alpha>0\}}$ be the primal-dual central path of (4.2) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. Let $\hat{Z}(\alpha)\in\mathbb{S}^{\hat{n}}$ be a principal submatrix of ${Z(\alpha)}$, for some $\hat{n}\in[r+1,n-1]$. Then for all $\alpha\in(0,\bar{\alpha})$ for some fixed $\bar{\alpha}>0$ it holds that,

[TABLE]

Proof.

By interlacing eigenvalues and Corollary 4.3 we have,

[TABLE]

The bound also holds for any other norm on $\mathbb{S}^{n}$. ∎

Now we show that after a suitable orthogonal transformation, ${Z(\alpha)}$ admits a block partition where at least $\operatorname{sd}(\mathcal{F})$ of these blocks converge to [math], each at a different rate.

Theorem 4.7**.**

Let ${\{({X(\alpha)},{y(\alpha)},{Z(\alpha)}):\alpha>0\}}$ be the primal-dual central path of (4.2) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. Let $\bar{\alpha}>0$ be fixed. Then there exists an integer ${d\in[\operatorname{sd}(\mathcal{F}),\bar{m}]}$ where,

[TABLE]

and a suitable orthogonal transformation of $\mathcal{F}$, such that,

[TABLE]

where for all $\alpha\in(0,\bar{\alpha})$ it holds that,

1. $(i)$

$Z_{1}(\alpha)\to S_{1}\succ 0$* and $Z_{i}(\alpha)\to 0$ for all $i\in\{2,\dotso,d+1\}$,* 2. $(ii)$

$\frac{\lambda_{\min}(Z_{i}(\alpha))}{\lambda_{\max}(Z_{i+1}(\alpha))}\to\infty$* for all $i\in\{1,\dotso,d\}$,* 3. $(iii)$

$\lambda_{\min}(Z_{i}(\alpha))=\Theta\left(\lambda_{\max}(Z_{i}(\alpha))\right)$* for all $i\in\{1,\dotso,d+1\}$,* 4. $(iv)$

$\lVert Z_{d+1}(\alpha)\rVert=\Theta(\alpha)$.

Proof.

We assume, without loss of generality, that,

[TABLE]

As above, let $(\bar{X},\bar{y},\bar{Z})$ be the limit point of the primal-dual central path. We know that $\bar{Z}$ is an exposing vector for a face containing $\operatorname{face}(\mathcal{F})$. Thus ${\bar{y}^{T}b=0}$. Without loss of generality we may assume that,

[TABLE]

Now we define $y^{1}:=\bar{y}$ and choose ${v^{1},\dotso,v^{m_{v}}\in\operatorname{{span}}\{b,y^{1}\}^{\perp}}$ for some ${m_{v}\leq m}$ so that the collection of vectors $\{y^{1},v^{1},\dotso,v^{m_{v}}\}$ is a basis for $b^{\perp}$. Note that when $b\neq 0$, then $\{b,y^{1},v^{1},\dotso,v^{m_{v}}\}$ is also a basis for ${\mathbb{R}^{m}\,}$.

Now for each ${\alpha>0}$ there exist coefficients $\beta(\alpha)$ and ${\nu_{1}(\alpha),\dotso,\nu_{m_{v}}(\alpha)}$ and $\gamma_{1}(\alpha)$ such that,

[TABLE]

Since ${y(\alpha)}\to y^{1}$ we have $\gamma_{1}(\alpha)\to 1$ and $\gamma_{1}(\alpha)$ dominates the other coefficients. That is,

[TABLE]

Now let us consider a block partition of ${Z(\alpha)}$ according to the block partition of $\bar{Z}$ in (4.7). We have,

[TABLE]

Note that the two diagonal blocks of ${Z(\alpha)}$ in (4.10) possess properties i and ii.

Let $Z_{11}(\alpha)$ denote the upper left block of ${Z(\alpha)}$. We consider two possibilities. First, suppose that $\lVert Z_{11}(\alpha)\rVert={\mathcal{O}}(\alpha)$. In this case these two diagonal blocks also satisfy properties iii and iv and we let $d=1$. That $d\leq\bar{m}$ is easy to see. Next, Lemma 4.6 implies that $Z_{11}(\alpha)$ has at most $r$ rows. Moreover, by our assumption on the facial structure of $\operatorname{face}(\mathcal{F})$ in (4.6) we conclude that $Z_{11}(\alpha)$ has exactly $r$ rows, otherwise $\bar{Z}$ exposes a face that is strictly smaller than $\operatorname{face}(\mathcal{F})$. Thus we have $\operatorname{sd}(\mathcal{F})=1\leq d$, as desired.

The second possibility is that $\lVert Z_{11}(\alpha)\rVert\neq{\mathcal{O}}(\alpha)$. In this case, at least one of the coefficients other than $\gamma_{1}(\alpha)$ converges to [math] at a rate not equal to ${\mathcal{O}}(\alpha)$. This coefficient is not $\beta(\alpha)$, since Lemma 4.5 implies that ${\beta(\alpha)=\Theta(\alpha)}$ when $b\neq 0$. When $b=0$ we may set $\beta(\alpha)=0$ as it is irrelevant. Thus we conclude that $\lvert\nu_{i}(\alpha)\rvert\neq{\mathcal{O}}(\alpha)$ for some $i\in\{1,\dotso,m_{v}\}$.

Now let, $y^{2}$ be a limit point of $\beta(\alpha)b+\sum_{i=1}^{m_{v}}\nu_{i}(\alpha)v^{i}$, after normalizing. By the arguments above, ${y^{2}\in\operatorname{{span}}\{v^{1},\dotso,v^{m_{v}}\}}$ and thus ${(y^{2})^{T}b=0}$. Secondly, $({\mathcal{A}}^{*}(\beta(\alpha)b+\sum_{i=1}^{m_{v}}\nu_{i}(\alpha)v^{i}))_{11}$ is positive definite for every $\alpha>0$ by (4.10). This implies that ${({\mathcal{A}}^{*}(y^{2}))_{11}\succeq 0}$. Thus if $({\mathcal{A}}^{*}(y^{2}))_{11}$ is not the zero matrix it is an exposing vector in the second step of facial reduction.

Let us first address the case ${({\mathcal{A}}^{*}(y^{2}))_{11}=0}$. Here we let $w^{1}:=y^{2}$ and choose

[TABLE]

for some $m_{v}$, different than the previously used $m_{v}$, so that ${\{y^{1},w^{1},v^{1},\dotso,v^{m_{v}}\}}$ is a basis for $b^{\perp}$. Now we repeat the above process until we obtain a new $y^{2}$ such that ${({\mathcal{A}}^{*}(y^{2}))_{11}\neq 0}$, i.e., we are in the second case.

Now we may assume that we have obtained $y^{2}$ as above and ${({\mathcal{A}}^{*}(y^{2}))_{11}\neq 0}$. We also assume, without loss of generality that,

[TABLE]

Then the matrix,

[TABLE]

exposes a face containing $\operatorname{face}(\mathcal{F})$ and this face is smaller than the one exposed by $\bar{Z}$. In other, words, we have obtained a better exposing vector. Let us assume that we have accumulated $m_{w}$ vectors of the type $w^{1}$ obtained in the case ${({\mathcal{A}}^{*}(y^{2}))_{11}=0}$. Then we choose,

[TABLE]

so that $\{y^{1},y^{2},w^{1},\dotso,w^{m_{w}},v^{1},\dotso,v^{m_{v}}\}$ is a basis for $b^{\perp}$. As above, there exist coefficients,

[TABLE]

such that,

[TABLE]

Then,

[TABLE]

where the upper left block is ${\mathcal{A}}^{*}_{11}\left(\beta(\alpha)b+\sum_{i=1}^{m_{v}}\nu_{i}(\alpha)v^{i}\right)$. By construction we have

[TABLE]

Thus we conclude that the diagonal blocks of ${Z(\alpha)}$ satisfy properties i and ii. In addition, the blocks containing $\gamma_{1}(S_{1})$ and $\gamma_{2}(S_{2})$ satisfy property iii.

By Lemma 4.6 we may continue in this fashion until,

[TABLE]

for some positive integer $d$ and the upper left block has norm that is ${\mathcal{O}}(\alpha)$. By reasoning analogous to that of the discussion following (4.10), we conclude that the blocks of ${Z(\alpha)}$, according to the block partition of (4.17), satisfy properties i - iv and that $d\in[\operatorname{sd}(\mathcal{F}),\bar{m}]$, as desired. ∎

The issue with different rates of convergence among the blocks of ${Z(\alpha)}$ that vanish, is one of a practical nature. Suppose a path-following algorithm is applied and accurately follows the primal-dual central path. Then, once the the block of ${Z(\alpha)}$ that converges to [math] at the fastest rate, $Z_{d+1}(\alpha)$, reaches machine precision, the remaining blocks cannot be made smaller. Hence the forward error is a function of the difference between the rate of convergence of the slowest block, $Z_{2}(\alpha)$, and the fastest block, $Z_{d+1}(\alpha)$.

The integer $d$ of Theorem 4.7 actually provides an upper bound on $\operatorname{sd}(\mathcal{F})$ that complements the lower bound of Theorem 3.11. However, this upper bound is generally intractable due to its reliance on an unknown orthogonal transformation. If the statement of the theorem can be translated to a statement about convergence rates of blocks of eigenvalues, then we would have a tractable upper bound on $\operatorname{sd}(\mathcal{F})$. However this may not be true as illustrated by the parametric sequence,

[TABLE]

Here $S(\alpha)$ has different rates of convergence among the diagonal, but the two eigenvalues that vanish, do so at the same rate, $\Theta(\alpha^{3})$. One way to guarantee that the diagonal blocks correspond to blocks of eigenvalues is the following.

Corollary 4.8**.**

Let ${\{({X(\alpha)},{y(\alpha)},{Z(\alpha)}):\alpha>0\}}$ be the primal-dual central path of (4.2) for a spectrahedron $\mathcal{F}=\mathcal{F}({\mathcal{A}},b)$ satisfying Assumption 4.1. Let $\bar{\alpha}>0$ be fixed and let $d$ be as in Theorem 4.7. For every $\alpha\in(0,\bar{\alpha})$ we assume that ${Z(\alpha)}$ has the block structure of Theorem 4.7. If every principal submatrix of ${Z(\alpha)}$ of the form,

[TABLE]

satisfies $\lambda_{\min}(S_{i}(\alpha))=\Theta\left(\lambda_{\min}(Z_{i}(\alpha))\right)$, then there are exactly $d$ different rates of convergence among the eigenvalues of ${X(\alpha)}$ that vanish.

Proof.

Applying the interlacing eigenvalue theorem with the principal submatrices,

[TABLE]

yields the upper bound of $\lambda_{\max}(Z_{i}(\alpha))$ and the lower bound $\lambda_{\min}(Z_{i}(\alpha))$ on a block of $\lambda({Z(\alpha)})$ having the same size as the number of rows $Z_{i}(\alpha)$. Since $\lambda_{\min}(Z_{i}(\alpha))=\Theta\left(\lambda_{\max}(Z_{i}(\alpha))\right)$, these bounds are the same. Thus we conclude that for each $i\in\{2,\dotso,d+1\}$ there is a block of eigenvalues that converges to [math] at the same rate as $Z_{i}(\alpha)$ does. The desired result follows from the relation ${X(\alpha)}=\alpha{Z(\alpha)}^{-1}$. ∎

The challenge for this result is that the hypothesis is unverifiable just as the block structure of ${Z(\alpha)}$ is unobservable without knowledge of the appropriate orthogonal transformation. However, our numerical observations indicate that the conclusion of the corollary holds for instances where singularity degree is known.

5 Numerical Case Studies

In this section we apply the main results above to obtain bounds on forward error, singularity degree, and maximum rank for several problems from the literature. Our analysis is focused on problems with larger singularity degree, although, we do study one instance with singularity degree $1$, in order to demonstrate ‘good’ convergence. For some of the instances, the exact singularity degree is known, allowing us to test the quality of our bounds. In other instances, the singularity degree is not known and we use our bounds to provide an estimate of it.

In order to study the notion that large singularity degree is sufficient, in some sense, for slow convergence, we consider bounded spectrahedra, satisfying Assumption 4.1. We follow the primal-dual central path of (4.2) with a path-following algorithm based on the Gauss-Newton search direction, see [15, 8].

For each problem, we present two plots based on a sequence of the type $\{\sigma^{k}\}$, as in Section 3, where $\sigma=0.6$. The first is the $Q$-convergence ratio as in Theorem 3.6. To make the subsequent discussion less cumbersome, we introduce the notation,

[TABLE]

We use this first ratio to bound maximum rank (Corollary 3.8), forward error (Theorem 3.10), and singularity degree (Theorem 3.11).

The second plot is the ratio of adjacent eigenvalues as in (3.3),

[TABLE]

This ratio is used to upper bound the maximum rank as in Proposition 3.3 and to determine the number of different rates of convergence among eigenvalues that vanish.

For each value of $k$, we approximate the ratios $R_{Q}(i,k)$ and $R_{N}(i,k)$ by following the primal-dual central path $({X(\alpha)},{y(\alpha)},{Z(\alpha)})$ of (4.2) to the point where $\alpha=\sigma^{k}$. Once the ratios have been obtained for $k$ sufficiently large, around $60$, we generate plots of the ratios against $k$ for each $i$ and obtain bounds on maximum rank, forward error, and singularity degree. The bounds as well as the true values (when available) are recorded in Table 5.1.

We denote the first spectrahedron as spec1 and it is taken from the class of SDPs introduced in [29]. There, the authors present a method to generate SDPs with specified complementarity gap. By considering the optimal set of one such SDP with strict complementarity, we obtain a spectrahedron with singularity degree $1$. The problem we consider has size $n=20$ and plots of the ratios $R_{Q}(i,k)$ and $R_{N}(i,k)$ are shown in Figure 5.1. In the left image, there is a clear distinction between curves that converge to $1$ and curves that do not converge to $1$. Moreover, if we disregard the irregularity in the last few values of the curves that do not converge to $1$, we may conclude that those curves converge to the smallest dashed line located at $0.6$. This observation, together with Theorem 4.4, correctly indicates that the spectrahedron has singularity degree $1$. Exactly $13$ of the curves converge to $0.6$, yielding $\overline{r}=7$, the correct approximation of the maximal rank $r$. The plot on the right side of the figure shows that exactly one curve blows up and it is the curve corresponding to $i=7=r$. This indicates, as expected, two groups of eigenvalues of $X$: those that converge to positive values and those that vanish.

The second spectrahedron, denoted spec2, is the classical ‘worst case’ problem as presented in [26], with $n=5$ and singularity degree $n-1=4$. Plots of the two ratios are in Figure 5.2. The left image shows $5$ distinct rates of $Q$-convergence, one for each eigenvalue. All but one of the curves converge to values that are clearly below $1$. This indicates, correctly, that the maximum rank is at most $1$. The largest of the curves appears to converge to the highest of the dashed lines. Thus we may infer that singularity degree is at least $4$. Since $4=n-1$, the worst case upper bound, we may conclude that singularity degree is exactly $4$. The row corresponding to spec2 in Table 5.1 shows a very large discrepancy between forward error and backward error. Our lower bound is actually quite close to the true forward error. Now let us consider the image on the right. It may be somewhat speculative to assert that the two lower curves blow up. Thus, taking the more cautious approach we assume that only the two larger curves blow up. Checking the indices of these curves yields an upper bound of $3$ on the maximum rank. We choose the notably lower estimate of $1$ based on the left plot. On the other hand if we are to apply Corollary 4.8 then we would want an overestimate of the number of different rates of convergence among the eigenvalues of ${X(\alpha)}$ that vanish. For this number we include the two lower curves, giving a bound of $4$.

For the next spectrahedron, spec3, the authors of [28] observed “strange behaviour” when attempting to optimize over it with an interior point method. The dimension is $n=10$ and the singularity degree is proven to be $5$. In Table 5.1 we see a large discrepancy between forward error and backward error. The left image of Figure 5.3 shows six distinct groups of curves. It is clear, for all but two of the curves, that the limit point is different from $1$. Thus we have an upper bound of $2$ on the maximum rank. This upper bound gives quite an accurate lower bound on the forward error. The largest of the curves that does not converge to $1$ appears to converge to a value that is below the fourth dashed line, indicating a lower bound of $4$ on the singularity degree. Unlike the two previous spectrahedra, here the lower bound on singularity degree is a strict one. The image on the right shows exactly five different rates of convergence among the eigenvalues of ${X(\alpha)}$ that converge to [math]. Moreover the upper bound on maximum rank corresponds to the one obtained from the left image.

The fourth spectrahedron, spec4, is generated by the algorithm of [29], just as spec1 is. However, for this instance we require the existence of a complementarity gap, i.e., strict complementarity does not hold. While we do not know the exact value of the singularity degree, the lack of strict complementarity gives us a lower bound of $2$. Plots of the ratios $R_{Q}(i,k)$ and $R_{N}(i,k)$ are shown in Figure 5.4. From the left image we can be quite sure that those curves that converge to the second dashed line or below do not converge to $1$. A closer inspection reveals that there are $10$ such curves, implying an upper bound of $5$ on the maximum rank. The corresponding lower bound on forward error, as recorded in Table 5.1, is indeed a lower bound and more informative than the reported backward error. We also obtain a lower bound on the singularity degree that coincides with the theoretical lower bound of $2$. The image on right shows exactly two rates of convergence among eigenvalues of ${X(\alpha)}$ that converge to [math] and, once again, provides the same upper bound on maximum rank as obtained from the image on left.

The final spectrahedron we consider is a Toeplitz cycle completion problem of the form introduced in Corollary 6 of [1]. The specific instance we consider here is that of Example 4.5 of [24] with $n=10$. While the singularity degree for this problem is not known, a lower bound of $2$ was proven in [24]. In Figure 5.5, we find images of plots of the ratios $R_{Q}(i,k)$ and $R_{N}(i,k)$. The left image indicates that all but two of the eigenvalues of ${X(\alpha)}$ converge to [math], yielding an exact approximation of maximum rank. Moreover, the corresponding eight curves appear to have limits below the second dashed line. Hence we have a lower bound of $2$ on the singularity degree. This coincides with the theoretical lower bound.

In these case studies we have demonstrated the ability to upper bound maximum rank quite effectively. The resulting lower bound on forward error is of a much larger magnitude than backward error in all instances with the exception of spec1, where the singularity degree is $1$. We see this feature as quite useful, as it alerts practitioners that the proposed solution is of substantially lower accuracy than backward error indicates. For spectrahedra with known singularity degree, we have demonstrated that the lower bound is quite accurate. In the other cases, the lower bound is in agreement with the theoretical lower bound. Lastly, for these test cases (as well as for others we have tested), the value $N_{\lambda}$ seems to be an upper bound on singularity degree. Proving this, or demonstrating a counterexample, is an interesting topic for future research.