The Generalized Trust Region Subproblem: solution complexity and convex hull results

TL;DR

This paper provides a convex hull characterization and a linear-time algorithm for solving the Generalized Trust Region Subproblem (GTRS) with nonconvex quadratic constraints, improving computational efficiency and extending to variants.

Contribution

It offers an explicit convex hull description of the GTRS set and develops a linear-time algorithm with error guarantees, applicable to various GTRS variants.

Findings

Convex hull characterized via generalized eigenvalues.

Linear-time algorithm for epsilon-approximate solutions.

Applicability to multiple GTRS variants with improved efficiency.

Abstract

We consider the Generalized Trust Region Subproblem (GTRS) of minimizing a nonconvex quadratic objective over a nonconvex quadratic constraint. A lifting of this problem recasts the GTRS as minimizing a linear objective subject to two nonconvex quadratic constraints. Our first main contribution is structural: we give an explicit description of the convex hull of this nonconvex set in terms of the generalized eigenvalues of an associated matrix pencil. This result may be of interest in building relaxations for nonconvex quadratic programs. Moreover, this result allows us to reformulate the GTRS as the minimization of two convex quadratic functions in the original space. Our next set of contributions is algorithmic: we present an algorithm for solving the GTRS up to an epsilon additive error based on this reformulation. We carefully handle numerical issues that arise from inexact generalized eigenvalue and eigenvector computations and establish explicit running time guarantees for these algorithms. Notably, our algorithms run in linear (in the size of the input) time. Furthermore, our algorithm for computing an epsilon-optimal solution has a slightly-improved running time dependence on epsilon over the state-of-the-art algorithm. Our analysis shows that the dominant cost in solving the GTRS lies in solving a generalized eigenvalue problem -- establishing a natural connection between these problems. Finally, generalizations of our convex hull results allow us to apply our algorithms and their theoretical guarantees directly to equality-, interval-, and hollow- constrained variants of the GTRS. This gives the first linear-time algorithm in the literature for these variants of the GTRS.

Figures

Equations

$$\mathrm{Opt}:=\underset{x\in {\mathbb{R}}^n}{\inf }\left\{q_0(x):\text{leavevmode}{q}_1(x)\le 0\right\},$$

$$\tilde{O}\left(\frac{N\sqrt{{\kappa }_{\text{HK}}}}{\sqrt{ϵ}}\log \left(\frac{n}{p}\right)\log \left(\frac{{\kappa }_{\text{HK}}}{ϵ}\right)\right),$$

$$\tilde{O}\left(\frac{N{\varphi }^3}{\sqrt{ϵ{\xi }_{\text{JL}}^5}}\log \left(\frac{n}{p}\right)\log {\left(\frac{\varphi }{ϵ{\xi }_{\text{JL}}}\right)}^2\right),$$

$$\mathrm{Opt}=\underset{(x,t)}{\inf }\left\{t:(x,t)\in \mathcal{S}\right\},$$

$$\displaystyle{\cal S}\coloneqq\left\{(x,t)\in{\mathbb{R}}^{n+1}:\,\begin{array}[]{l}q_{0}(x)\leq t\\ q_{1}(x)\leq 0\end{array}\right\}.$$

$$\mathrm{Opt}=\underset{x,t}{\inf }\left\{t:\text{leavevmode}(x,t)\in \mathrm{conv}(\mathcal{S})\right\}=\underset{x,t}{\inf }\left\{t:\text{leavevmode}(x,t)\in \overline{\mathrm{conv}}(\mathcal{S})\right\}.$$

$$\tilde{O}\left(\frac{N{\kappa }^{3/2}\sqrt{\zeta }}{\sqrt{ϵ}}\log \left(\frac{n}{p}\right)\log \left(\frac{\kappa }{ϵ}\right)\right),\tilde{O}\left(\frac{N{\kappa }^2\sqrt{\zeta }}{\sqrt{ϵ}}\log \left(\frac{n}{p}\right)\log \left(\frac{\kappa }{ϵ}\right)\right),$$

$$O\left(\frac{N\kappa \sqrt{\zeta }}{\sqrt{ϵ}}\right).$$

$$q(\gamma ,x):=q_0(x)+\gamma q_1(x).$$

$$\mathcal{S}(\gamma ):=\left\{(x,t)\in {\mathbb{R}}^{n+1}:\text{leavevmode}q(\gamma ,x)\le t\right\}.$$

$$\displaystyle{\cal S}\coloneqq\left\{(x,t)\in{\mathbb{R}}^{n+1}:\leavevmode\nobreak\ \begin{array}[]{l}q_{0}(x)\leq t\\ q_{1}(x)\leq 0\end{array}\right\}=\bigcap_{\gamma\geq 0}{\cal S}(\gamma).$$

$$\Gamma :=\left\{\gamma \in {\mathbb{R}}_{+}:\text{leavevmode}A(\gamma )⪰0\right\}.$$

$$\displaystyle\mathfrak{S}\coloneqq\left\{(x,t)\in{\mathbb{R}}^{n+1}:\leavevmode\nobreak\ \begin{array}[]{l}q_{0}(x)\leq t\\ q_{1}(x)=0\end{array}\right\}.$$

$$\mathrm{conv}(\mathcal{S})=\mathrm{conv}(\mathfrak{S})=\mathcal{S}({\gamma }_{-})\cap \mathcal{S}({\gamma }_{+}).$$

$$\underset{x\in {\mathbb{R}}^n}{\min }\left\{q_0(x):\text{leavevmode}{q}_1(x)\le 0\right\}=\underset{x\in {\mathbb{R}}^n}{\min }\max \left\{q({\gamma }_{-},x),q({\gamma }_{+},x)\right\}.$$

$$\overline{\mathrm{conv}}(\mathcal{S})=\overline{\mathrm{conv}}(\mathfrak{S})=\mathcal{S}({\gamma }_{-})\cap \mathcal{S}({\gamma }_{+}).$$

$$\underset{x\in {\mathbb{R}}^n}{\inf }\left\{q_0(x):\text{leavevmode}{q}_1(x)\le 0\right\}=\underset{x\in {\mathbb{R}}^n}{\inf }\max \left\{q({\gamma }_{-},x),q({\gamma }_{+},x)\right\}.$$

$$\mathcal{S}(\gamma )$$

$$=\left\{(x,t)\in {\mathbb{R}}^{n+1}:\text{leavevmode}{x}^{\top }A(\gamma )x+2b(\gamma {)}^{\top }x+c(\gamma )\le t\right\}$$

$$q({\gamma }_{+},{\hat{x}}_{\alpha })-{\hat{t}}_{\alpha }$$

$$=\left(q({\gamma }_{+},\hat{x})-\hat{t}\right)+2\alpha \left({\hat{x}}^{\top }A({\gamma }_{+})d+b({\gamma }_{+}{)}^{\top }d-e/2\right)+{\alpha }^2{d}^{\top }A({\gamma }_{+})d$$

$$=q({\gamma }_{+},\hat{x})-\hat{t},$$

$$q({\gamma }_{-},{\hat{x}}_{\alpha })-{\hat{t}}_{\alpha }$$

$$=\left(q({\gamma }_{-},\hat{x})-\hat{t}\right)+2\alpha \left({\hat{x}}^{\top }A({\gamma }_{-})d+b({\gamma }_{-}{)}^{\top }d-e/2\right)+{\alpha }^2{d}^{\top }A({\gamma }_{-})d.$$

$$q({\gamma }_{-},\hat{x})=q_0(\hat{x})+{\gamma }_{-}{q}_1(\hat{x})<q_0(\hat{x})+{\gamma }_{+}{q}_1(\hat{x})=q({\gamma }_{+},\hat{x}),$$

$$q_1({\hat{x}}_{{\alpha }_i})$$

$$q_0({\hat{x}}_{{\alpha }_i})$$

$$q({\gamma }_{-},{\hat{x}}_{\alpha })-{\hat{t}}_{\alpha }$$

$$=\left(q({\gamma }_{-},\hat{x})-\hat{t}\right)+2\alpha \left({\hat{x}}^{\top }A({\gamma }_{-})d+b({\gamma }_{-}{)}^{\top }d-e/2\right)+{\alpha }^2{d}^{\top }A({\gamma }_{-})d$$

$$=q({\gamma }_{-},\hat{x})-\hat{t}.$$

$$q({\gamma }_{+},{\hat{x}}_{\alpha })-{\hat{t}}_{\alpha }$$

$$=\left(q({\gamma }_{+},\hat{x})-\hat{t}\right)+2\alpha \left({\hat{x}}^{\top }A({\gamma }_{+})d+b({\gamma }_{+}{)}^{\top }d-e/2\right)+{\alpha }^2{d}^{\top }A({\gamma }_{+})d.$$

$$\mathcal{S}({\gamma }_{-})\cap \mathcal{S}({\gamma }_{+})\subseteq \mathrm{conv}(\mathfrak{S})\subseteq \mathrm{conv}(\mathcal{S})\subseteq \mathcal{S}({\gamma }_{-})\cap \mathcal{S}({\gamma }_{+}).$$

$$\underset{(x,t)\in {\mathbb{R}}^{n+1}}{\inf }\left\{t:\text{leavevmode}(x,t)\in \mathcal{S}\right\}$$

Full text

1 Introduction

In this paper, we study the Generalized Trust-Region Subproblem (GTRS), which is defined as

[TABLE]

where $q_{0}:{\mathbb{R}}^{n}\to{\mathbb{R}}$ and $q_{1}:{\mathbb{R}}^{n}\to{\mathbb{R}}$ are general quadratic functions of the form $q_{i}(x)=x^{\top}A_{i}x+2b_{i}^{\top}x+c_{i}$. Here, $A_{i}\in{\mathbb{R}}^{n\times n}$ are symmetric matrices, $b_{i}\in{\mathbb{R}}^{n}$ and $c_{i}\in{\mathbb{R}}$. We are interested, in particular, in the case where $q_{0}$ and $q_{1}$ are both nonconvex, i.e., $A_{i}$ has at least one negative eigenvalue for both $i=0,1$.

Problem (1), introduced and studied by Moré [25], Stern and Wolkowicz [33], generalizes the classical Trust-Region Subproblem (TRS) [6] in which one is asked to optimize a nonconvex quadratic objective over a Euclidean ball. The TRS is an essential ingredient of trust-region methods that are commonly used to solve continuous nonconvex optimization problems [6, 28, 30] and also arises in applications such as robust optimization [2, 14]. On the other hand, the GTRS has applications in nonconvex quadratic integer programs, signal processing, and compressed sensing; see [4, 1, 18] and references therein for more applications.

Although the TRS, as stated, is nonlinear and nonconvex, it is well-known that its semidefinite programming (SDP) relaxation is exact. Consequently, the TRS and a number of its variants can be solved in polynomial time via SDP-based techniques [31, 9] or using specialized nonlinear algorithms [11, 26]. In fact, custom iterative methods with linear (in the size of the input) running times have been shown in a few works. Hazan and Koren [13] proposed an algorithm to solve the TRS (as well as the GTRS when $A_{1}$ is positive definite) based on repeated approximate eigenvector computations. This algorithm runs in time

[TABLE]

where $N$ is the number of nonzero entries in the matrices $A_{0}$ and $A_{1}$, $\epsilon$ is the additive error, $n$ is the dimension of the problem, $p$ is the failure probability, and $\kappa_{\text{HK}}$ is a condition number. This was the first algorithm in the literature shown to achieve a linear time complexity. Here, and in the remainder of the paper, the term “linear” is used to describe running times that scale at most linearly with $N$ but may depend arbitrarily on its other parameters. Afterwards, Ho-Nguyen and Kılınç-Karzan [15] presented another linear-time algorithm for the TRS with a slightly better overall complexity, eliminating the $\mathop{{\rm log}}(\kappa_{\text{HK}}/{\epsilon})$ term. Their approach reformulates the TRS as minimizing a convex quadratic objective over the Euclidean ball, and solving the resulting smooth convex optimization problem via Nesterov’s accelerated gradient descent method. In contrast to [13], this convex reformulation approach requires only a single minimum eigenvalue computation. Wang and Xia [35] also suggested using Nesterov’s algorithm in the case of the interval-constrained TRS.

The GTRS shares a number of nice properties of the TRS. For example, by the S-lemma, it is well-known that the GTRS also admits an exact SDP reformulation under the Slater condition [10, 29]. Thus, while quadratically-constrained quadratic programming is NP-hard in general, there are polynomial-time SDP-based algorithms for solving the GTRS. Nevertheless, the relatively large computational complexity of SDP-based algorithms prevents them from being applied as a black box to solve large-scale instances of the GTRS. A variety of custom approaches have been developed to solve the GTRS; for earlier work on this domain see [25, 33, 8] and references therein.

One line of work has developed algorithms for solving the GTRS when the matrices $A_{0}$ and $A_{1}$ are simultaneously diagonalizable (SD) (see Jiang and Li [16] and references therein for background on the SD condition). Under the SD condition, along with certain restrictions on the quadratics $q_{0}$ and $q_{1}$, Ben-Tal and Teboulle [3] provide a reformulation of the interval-constrained GTRS as a convex minimization problem with linear constraints. More recently, Ben-Tal and den Hertog [2] show that there is a second order cone programming (SOCP) reformulation of the GTRS in a lifted space under the SD condition. Subsequent work by Locatelli [22] extends Ben-Tal and den Hertog [2] by illustrating some additional settings in which the SOCP reformulation is tight. Under the SD condition, Fallahi et al. [7] exploit the separable structure of the problem and, using Lagrangian duality, they suggest a solution procedure based on solving a univariate convex minimization problem. Salahi and Taati [32] derive an algorithm for solving the interval-constrained GTRS by exploiting the structure of the dual problem under the SD condition. By applying a simultaneous block diagonalization approach, Jiang et al. [19] generalize Ben-Tal and den Hertog [2] and provide an SOCP reformulation for the GTRS in a lifted space when the problem has a finite optimal value. Their methods apply even when $q_{0}$ and $q_{1}$ do not satisfy the SD condition. They further derive a closed-form solution when the SD condition fails and examine the case of interval- or equality-constrained GTRS. In this line of work, it is often assumed implicitly that $A_{0}$ and $A_{1}$ are already diagonal or that a simultaneously-diagonalizing basis can be computed. The only method that we know of for computing such a basis relies on exact matrix eigen-decomposition. Thus, although experiments have been presented [32, 19] suggesting that such algorithms (where exact procedures are replaced by numerical ones) may perform well, theoretical guarantees have yet to be established. Furthermore, the large cost of matrix eigen-decomposition prevents the application of these algorithms to large-scale instances of the GTRS.

A second line of work has explored the connections between the GTRS and generalized eigenvalues of the matrix pencil $A_{0}+\gamma A_{1}$. These works all assume a regularity condition about the matrix pencil: there exists a $\gamma\geq 0$ such that $A_{0}+\gamma A_{1}$ is either positive definite or positive semidefinite.111In fact, this assumption can be made without loss of generality; see Remark 1. Pong and Wolkowicz [30] study the optimality structure of the GTRS and propose a generalized-eigenvalue-based algorithm which exploits this structure. Unfortunately, an explicit running time is not presented in [30]. Adachi and Nakatsukasa [1] present another generalized-eigenvalue-based algorithm motivated by similar observations. The dominant costs present in this algorithm come from computing a pair of generalized eigenvalues and solving a linear system. Ignoring issues of exact computations, the runtime of this algorithm is $O(n^{3})$. Jiang and Li [17] show how to reformulate the GTRS as a convex quadratic program in terms of generalized eigenvalues. They establish that a saddle-point-based first-order algorithm can be used to solve the reformulation within an $\epsilon$ additive error in $O(1/\epsilon)$ time. In this line of work, it is often assumed that the generalized eigenvalues are given or can be computed exactly. In particular, theoretical guarantees have not yet been given regarding how these algorithms perform when only approximate generalized eigenvalue computations are available. This is of interest as, in practice, we cannot hope to numerically compute generalized eigenvalues exactly; see also the discussion in Section 4.1 in [18]. We would like to remark that numerical experiments in these papers [17, 1, 30] have suggested that algorithms motivated by these ideas may perform well even using only approximate generalized eigenvalue computations.

The very recent work of Jiang and Li [18] presents an algorithm for solving the GTRS up to an $\epsilon$ additive error in the objective with high probability under the regularity condition. This algorithm relies on machinery developed by [13] for solving the TRS and differs from previous algorithms in that it does not assume the ability to compute a simultaneously-diagonalizing basis or generalized eigenvalues. The running time of this algorithm is

[TABLE]

where $N$ is the number of nonzero entries in $A_{0}$ and $A_{1}$, $\epsilon$ is the additive error, $n$ is the dimension, $p$ is the failure probability, and $(\phi,\xi_{\text{JL}})$ are a pair of parameters measuring the regularity of the GTRS. In particular, this algorithm is able to take advantage of sparsity in the description of the quadratic functions. To our knowledge, this is the first provably linear-time algorithm for the GTRS to be presented in the literature.

In this paper, we derive a new algorithm for the GTRS based on a convex quadratic reformulation in the original space. This algorithm can also be applied to variants of the GTRS with interval, equality, or hollow constraints. The basic idea in our approach relies on the fact that we can provide exact (closed) convex hull characterizations of the epigraph of the GTRS. We summarize our results below and provide an outline of the paper.

1. (i)

We rewrite the GTRS with a linear objective

[TABLE]

where the set ${\cal S}$ is defined as

[TABLE]

As the objective in (4) is linear, we can take either the convex hull or closed convex hull of the feasible domain. Then,

[TABLE]

In Section 2, we give an explicit description of the set $\operatorname{conv}({\cal S})$ (respectively, $\operatorname{\overline{conv}}({\cal S})$). Specifically, we show that when the respective assumptions are satisfied, $\operatorname{conv}({\cal S})$ and $\operatorname{\overline{conv}}({\cal S})$ can both be described in terms of two convex quadratic functions determined by the generalized eigenvalue structure of the matrix pencil $A_{0}+\gamma A_{1}$. We note that these convex hull results may be of independent interest in building relaxations and/or algorithms for nonconvex quadratic programs with or without integer variables. As an immediate consequence of these (closed) convex hull results, we can reformulate the GTRS as the minimization of the maximum of two convex quadratics. This convex reformulation was previously discovered by Jiang and Li [17] by considering the Lagrangian dual and proving a zero duality gap. Our approach shows that the reformulation is tight for a very intuitive reason — the convex hull of the epigraph is exactly characterized by the convex quadratics used in the reformulation. 2. (ii)

The proofs in Section 2 actually imply stronger convex hull results: under the same assumptions, the (closed) convex hull of ${\cal S}$ is generated by points in ${\cal S}$ where the constraint $q_{1}(x)\leq 0$ is tight. This observation immediately leads to interesting consequences, which we detail in Section 3. Specifically, we extend our (closed) convex hull results to handle epigraph sets that arise when additional nonintersecting constraints are imposed on the GTRS. This will allow us to extend our algorithms to variants of the GTRS present in the literature [3, 2, 19, 17, 30, 32, 15, 36, 33, 25]. Specifically, this generalization allows us to handle interval-, equality-, and hollow-constrained GTRS. 3. (iii)

In Section 4, we give a careful analysis of the numerical issues that come up for an algorithm based on the above ideas. At a high level, we show that by approximating the generalized eigenvalues sufficiently well, the perturbed convex reformulation is within a small additive error of the true convex reformulation. Then, by leveraging the concavity of the function $\lambda_{\min}(A_{0}+\gamma A_{1})$, in the variable $\gamma$, we show how to approximate the necessary generalized eigenvalues efficiently. We believe this subroutine and the theoretical guarantees we present for it may also be of independent interest in other contexts. Next, we utilize an algorithm proposed by Nesterov [27, Section 2.3.3] for solving general minimax problems with smooth components to solve our convex reformulation with a convergence rate of $\tilde{O}(1/\sqrt{\epsilon})$. This contrasts the approach taken by Jiang and Li [17] that analyzes a saddle-point-based first-order algorithm and results in a convergence rate of $O(1/\epsilon)$. In order to apply the algorithm proposed by Nesterov, we establish that the gradient mapping step can be computed efficiently in our context. Finally, relying on our convex hull characterization, we show how to recover an approximate solution of the GTRS using only approximate eigenvectors.

We present two algorithms (Algorithms 1 and 4). The former finds an $\epsilon$-optimal value and the latter finds an $\epsilon$-optimal feasible solution. In other words, the former returns a scalar in $[\operatorname{Opt},\operatorname{Opt}+\epsilon]$ and the latter returns a vector $x$ in the feasible region with $q_{0}(x)\in[\operatorname{Opt},\operatorname{Opt}+\epsilon]$. Their running times are

[TABLE]

respectively. Here, $\xi$, $\zeta$, and $\kappa$ are regularity parameters of the matrix pencil $A_{0}+\gamma A_{1}$ (see Definition 1). Comparing (8) and (2), we see that our running times match the dependences on $N$, $n$, $\epsilon$, and $p$ from the algorithm for the TRS presented by Hazan and Koren [13]. Comparing (8) (specifically the running time for finding an $\epsilon$-optimal solution) and (3), we see that our running time matches the linear dependence on $N$ and improves the dependence on $\epsilon$ by a logarithmic factor from the running time presented by Jiang and Li [18]. The dependences on the regularity parameters in the two running times are incomparable (see Remark 8) but there exist examples where our running time gives a polynomial-order improvement upon the running time presented by Jiang and Li [18] (see Remark 12).

In comparison to the approach taken by Jiang and Li [18], we believe our approach is conceptually simpler and more straightforward to implement. In particular our approach directly solves the GTRS in the primal space as opposed to solving a feasibility version of the dual problem. Moreover, our analysis highlights the connection between the GTRS and generalized eigenvalue problems, and in fact demonstrates that the dominant cost in solving the GTRS is the cost of solving a generalized eigenvalue problem.

In our running times (8), the large dependence on the regularity parameters arises from the error that is introduced as a result of inexact generalized eigenvalue and eigenvector computations. We illustrate that our algorithms can be substantially sped up if we have access to exact generalized eigenvalue and eigenvector methods. In particular, we show that when $A_{0}$ and $A_{1}$ are diagonal, we can compute an $\epsilon$-optimal solution to the GTRS in time

[TABLE]

As mentioned previously, the generalizations of our convex hull results allow us to apply our algorithms to variants of the GTRS. In particular, our algorithms can be applied without change to interval-, equality-, or hollow-constrained GTRS.

Our study of the convex hull of the epigraph of GTRS is inspired by convex hull results in related contexts. The recent work of Ho-Nguyen and Kılınç-Karzan [15] gives a characterization on the convex hull of the epigraph of the TRS. In particular, under the assumption that $A_{1}$ is positive definite, Ho-Nguyen and Kılınç-Karzan [15, Theorem 3.5] give the explicit closed convex hull characterization of the set ${\cal S}$. In this respect, one can view our developments on the (closed) convex hull of ${\cal S}$ when neither $A_{0}$ nor $A_{1}$ is positive semidefinite as complementary to the results of Ho-Nguyen and Kılınç-Karzan [15, Section 3]. Notably, in contrast to [15, Section 3], we have to handle a number of issues that arise due to the recessive directions of the nonconvex domain. The papers by Yıldıran [37], Modaresi and Vielma [24] are also closely related to our convex hull results. Yıldıran [37] studies the convex hull of the intersection of two strict quadratic inequalities (note that the resulting set is open) under the milder regularity condition that there exists $\gamma\geq 0$ such that $A_{0}+\gamma A_{1}$ is positive semidefinite, and Modaresi and Vielma [24] analyze conditions under which one can safely take the closure of the sets in Yıldıran [37] and still obtain the desired closed convex hull results. In contrast, our analysis leverages the additional structure present in an epigraph set to give a more direct proof of the convex hull result. Furthermore, as our analysis is constructive (given $x\in\operatorname{conv}({\cal S})$, we show how to find two points $x_{1}$, $x_{2}\in{\cal S}$ such that $x\in[x_{1},x_{2}]$), it immediately suggests a rounding procedure (given a solution to the convex reformulation, we show how to find a solution to the original GTRS). This contrasts the analysis in Yıldıran [37], where such a rounding procedure is not obvious. Moreover, our analysis provides a more refined result that easily extends to variants of the GTRS with non-intersecting constraints. Finally, we would like to mention related work on convex hulls of sets defined by second-order cones (SOCs). Burer and Kılınç-Karzan [5] study the convex hull of the intersection of a convex and nonconvex quadratic or the intersection of an SOC with a nonconvex quadratic. Similarly, the convex hull of the a two-term disjunction applied to an SOC or its cross section has received much attention (see [5, 20] and references therein). As our focus has been on the case where neither $A_{0}$ nor $A_{1}$ is positive semidefinite, we view our developments as complementary to these results.

Notation. Given a symmetric matrix, $A\in{\mathbb{R}}^{n\times n}$, let $\lambda_{\min}(A)$ and $\lambda_{\max}(A)$ denote the minimum and maximum eigenvalues of $A$. We write $A\succeq 0$ (respectively, $A\succ 0$) if $A$ is positive semidefinite (respectively, positive definite). Let $\left\lVert A\right\rVert$ denote the spectral norm of $A$, i.e. $\left\lVert A\right\rVert=\max\left\{\left\lvert\lambda_{\min}(A)\right\rvert,\left\lvert\lambda_{\max}(A)\right\rvert\right\}$. Let $\det(A)$ and $\operatorname{tr}(A)$ denote the determinant and trace of $A$. For $a\in{\mathbb{R}}^{n}$, let $\operatorname{Diag}(a)$ denote the diagonal matrix $A\in{\mathbb{R}}^{n\times n}$ with diagonal entries $A_{i,i}=a_{i}$. Let $I_{n}$ be the $n\times n$ identity matrix. For a set $S\subseteq{\mathbb{R}}^{n}$, let $\operatorname{conv}(S)$ and $\operatorname{\overline{conv}}(S)$ be the convex hull and closed convex hull of $S$, respectively. For $x\in{\mathbb{R}}^{n}$, let $\left\lVert x\right\rVert$ be its Euclidean norm. For $x\in{\mathbb{R}}^{n}$ and $r\geq 0$, let $B(x,r)$ be the closed ball of radius $r$ centered at $x$, i.e., $B(x,r)=\left\{y\in{\mathbb{R}}^{n}:\,\left\lVert x-y\right\rVert\leq r\right\}$. Let ${\mathbb{R}}_{+}$ denote the nonnegative reals. Let $\nabla$ denote the gradient operator. We will use $\tilde{O}$-notation to hide $\mathop{{\rm log}}\mathop{{\rm log}}$-factors in running times.

2 Convex hull characterization

In this section we discuss our (closed) convex hull results. We will aggregate the objective function $q_{0}$ with the constraint $q_{1}$ using a nonnegative aggregation weight to derive relaxations of the set ${\cal S}$. We then show that under a mild assumption the (closed) convex hull of ${\cal S}$ can be described by two convex quadratic functions obtained from this aggregation technique.

Let $q:{\mathbb{R}}\times{\mathbb{R}}^{n}\to{\mathbb{R}}$ be defined as

[TABLE]

Let $A:{\mathbb{R}}\to{\mathbb{R}}^{n\times n}$ be defined as $A(\gamma)\coloneqq A_{0}+\gamma A_{1}$. Similarly define $b(\gamma)$ and $c(\gamma)$. In particular, $q(\gamma,x)=x^{\top}A(\gamma)x+2b(\gamma)^{\top}x+c(\gamma)$. We stress that while $q(0,x)=q_{0}(x)$, we have $q(1,x)=q_{0}(x)+q_{1}(x)$ which is not equal to $q_{1}(x)$ in general.

Note that $q(\gamma,x)$ is linear in its first argument and quadratic in its second argument. This structure plays a large role in our analysis.

In order to derive valid relaxations to ${\cal S}$ based on aggregation, we will consider only nonnegative $\gamma$ in the remainder of the paper. For $\gamma\geq 0$, define

[TABLE]

Note that ${\cal S}\subseteq{\cal S}(\gamma)$ holds for all $\gamma\geq 0$. Furthermore, it is clear that $q_{0}(x)\leq t$ and $q_{1}(x)\leq 0$ if and only if $q(\gamma,x)\leq t$ for all $\gamma\geq 0$. Thus, we can rewrite ${\cal S}$ as

[TABLE]

Note that the set ${\cal S}(\gamma)$ is convex if and only if $A(\gamma)\succeq 0$. We will define $\Gamma$ to be these $\gamma$ values, i.e.,

[TABLE]

Note that $\Gamma$ is a closed (possibly empty) interval. When this interval is nonempty, we will write it as $\Gamma=[\gamma_{-},\gamma_{+}]$.

We use the following two assumptions in our convex hull characterizations:

Assumption 1**.**

The matrices $A_{0}$ and $A_{1}$ both have negative eigenvalues and there exists a $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succ 0$.

Assumption 2**.**

The matrices $A_{0}$ and $A_{1}$ both have negative eigenvalues and there exists a $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succeq 0$.

Remark 1**.**

We claim that the case where $A_{0}$ and $A_{1}$ both have negative eigenvalues but do not satisfy either of the above assumptions is not interesting. In particular if $A_{0}$ and $A_{1}$ both have negative eigenvalues and $A(\gamma)\not\succeq 0$ for all $\gamma\geq 0$, then it is easy to show (apply the S-lemma then note that $A_{0}$ has a negative eigenvalue) that $\operatorname{conv}({\cal S})={\mathbb{R}}^{n+1}$. Consequently, the optimal value of the GTRS is always $-\infty$ in this case.

The assumption that there exists a $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succeq 0$ is made in most of the present literature on the GTRS [2, 1, 16, 17, 18, 30, 32] and convex hulls of the intersection of two quadratics [37, 24] either implicitly (for example, by assuming that an optimizer exists or that the optimal value is finite) or explicitly.

It is well-known that Assumption 1 implies that $A_{0}$ and $A_{1}$ are simultaneously diagonalizable. Even so, we will refrain from assuming that our matrices are diagonal and opt to work on a general basis. We choose to do this as the proofs of our convex hull results will serve as the basis for our algorithms, which do not have access to a simultaneously-diagonalizing basis.

Remark 2**.**

Assumptions 1 and 2 each imply that $\Gamma$ is nonempty and, consequently, that $\gamma_{-}$ and $\gamma_{+}$ exist. In addition, as $A(\gamma_{-})$ and $A(\gamma_{+})$ are both on the boundary of the positive semidefinite cone, they both have zero as an eigenvalue.

Under Assumption 1, the existence of some $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succ 0$ implies that $\gamma_{-}<\gamma^{*}<\gamma_{+}$ and hence $\gamma_{-}$ and $\gamma_{+}$ are distinct. Furthermore, as $\gamma^{*}\in(\gamma_{-},\gamma_{+})$, we have $d^{\top}A(\gamma_{-})d=d^{\top}A(\gamma_{+})d=0$ if and only if $d=0$.

In contrast, under Assumption 2, it is possible to have $\gamma_{-}=\gamma^{*}=\gamma_{+}$ and $\Gamma=\{\gamma^{*}\}$.

Finally, define $\mathfrak{S}$ to be the subset of ${\cal S}$ where the constraint $q_{1}(x)\leq 0$ is tight.

[TABLE]

When either Assumption 1 or 2 holds, $A_{1}$ has both positive and negative eigenvalues so that $\mathfrak{S}$ is nonempty.

We now state our (closed) convex hull results:

Theorem 1**.**

Under Assumption 1, we have

[TABLE]

In particular,

[TABLE]

Theorem 2**.**

Under Assumption 2, we have

[TABLE]

In particular,

[TABLE]

Remark 3**.**

The convex reformulation given in the second part of Theorem 2 was first proved by Jiang and Li [17] using a different argument without relying on the convex hull structure of the underlying sets. In contrast, the first part of Theorem 2 establishes a fundamental convex hull result highlighting the crux of why such a convex reformulation is possible.

We present the proof of Theorem 1 in Section 2.1. The proof of Theorem 2 is presented in Section 2.2 and relies on Theorem 1.

2.1 Proof of Theorem 1

Lemma 1**.**

The set ${\cal S}(\gamma)$ is convex and closed for all $\gamma\in\Gamma$.

Proof.

Let $\gamma\in\Gamma$ and recall the definition of ${\cal S}(\gamma)$.

[TABLE]

By the definition of $\Gamma$, we have $A(\gamma)\succeq 0$. Thus, the constraint defining ${\cal S}(\gamma)$ is convex in $(x,t)$, and we conclude that ${\cal S}(\gamma)$ is convex. Closedness of ${\cal S}(\gamma)$ follows by noting that it is the preimage of $(-\infty,0]$ under a continuous map. ∎

Lemma 2**.**

Suppose $\Gamma$ is nonempty and write $\Gamma=[\gamma_{-},\gamma_{+}]$. Then, $\operatorname{conv}({\cal S})\subseteq{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$.

Proof.

Note that ${\cal S}=\bigcap_{\gamma\geq 0}{\cal S}(\gamma)\subseteq{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. The result then follows by taking the convex hull of each side and noting that both ${\cal S}(\gamma_{-})$ and ${\cal S}(\gamma_{+})$ are convex by Lemma 1. ∎

The bulk of the work in proving Theorem 1 lies in the following result.

Lemma 3**.**

Under Assumption 1, we have ${\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})\subseteq\operatorname{conv}(\mathfrak{S})$.

Proof.

Let $(\hat{x},\hat{t})\in{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. We will show that $(\hat{x},\hat{t})\in\operatorname{conv}(\mathfrak{S})$. We split the analysis into three cases: (i) $q_{1}(\hat{x})=0$, (ii) $q_{1}(\hat{x})>0$, and (iii) $q_{1}(\hat{x})<0$.

1. (i)

If $q_{1}(\hat{x})=0$, then $q_{0}(\hat{x})=q_{0}(\hat{x})+\gamma_{-}q_{1}(\hat{x})=q(\gamma_{-},\hat{x})$. As $(\hat{x},\hat{t})\in{\cal S}(\gamma_{-})$ by assumption, we deduce that $q(\gamma_{-},\hat{x})\leq\hat{t}$. Combining these inequalities, we have that $q_{0}(\hat{x})=q(\gamma_{-},\hat{x})\leq\hat{t}$ and that $(\hat{x},\hat{t})\in\mathfrak{S}$. 2. (ii)

Now suppose $q_{1}(\hat{x})>0$. Let $d\neq 0$ such that $d^{\top}A(\gamma_{+})d=0$ (such a vector $d$ exists as $A(\gamma_{+})$ has zero as an eigenvalue; see Remark 2) and define $e\coloneqq 2\left(\hat{x}^{\top}A(\gamma_{+})d+b(\gamma_{+})^{\top}d\right)$. We modify $(\hat{x},\hat{t})$ along the direction $(d,e)$: For $\alpha\in{\mathbb{R}}$, let $(\hat{x}_{\alpha},\hat{t}_{\alpha})\coloneqq(\hat{x}+\alpha d,\hat{t}+\alpha e)$. We will show that there exist $\alpha_{1}<0<\alpha_{2}$ such that $(\hat{x}_{\alpha_{i}},\hat{t}_{\alpha_{i}})\in\mathfrak{S}$ for $i=1,2$, whence $(\hat{x},\hat{t})\in\operatorname{conv}(\mathfrak{S})$.

We study the behavior of the expressions $q(\gamma_{-},\hat{x}_{\alpha})-\hat{t}_{\alpha}$ and $q(\gamma_{+},\hat{x}_{\alpha})-\hat{t}_{\alpha}$ as functions of $\alpha$. A short calculation shows that for any $\alpha\in{\mathbb{R}}$, we have

[TABLE]

where the last equation follows from the definition of $e$. Thus, $q(\gamma_{+},\hat{x}_{\alpha})-\hat{t}_{\alpha}$ is constant in $\alpha$. Next, we compute

[TABLE]

As $d\neq 0$ and $d^{\top}A(\gamma_{+})d=0$, we deduce that $d^{\top}A(\gamma_{-})d\neq 0$ (see Remark 2). Then, as $A(\gamma_{-})\succeq 0$, we have that $d^{\top}A(\gamma_{-})d>0$. Hence, $q(\gamma_{-},\hat{x}_{\alpha})-\hat{t}_{\alpha}$ is strongly convex in $\alpha$.

Note that

[TABLE]

where the inequality follows from the fact that $\gamma_{-}<\gamma_{+}$ and $q_{1}(\hat{x})>0$. Therefore, $q(\gamma_{-},\hat{x})-\hat{t}<q(\gamma_{+},\hat{x})-\hat{t}$. Thus, there are values $\alpha_{1}<0<\alpha_{2}$ such that $q(\gamma_{-},\hat{x}_{\alpha_{i}})-\hat{t}_{\alpha_{i}}=q(\gamma_{+},\hat{x}_{\alpha_{i}})-\hat{t}_{\alpha_{i}}$ for $i=1,2$.

It remains to show that $(\hat{x}_{\alpha_{i}},\hat{t}_{\alpha_{i}})\in\mathfrak{S}$ for $i=1,2$. This follows immediately because for $i=1,2$, we have

[TABLE]

Then, applying (ii) and recalling that $q(\gamma_{+},\hat{x})\leq\hat{t}$, we have

[TABLE] 3. (iii)

The final case is symmetric to case (ii), thus we will only sketch its proof.

Suppose $q_{1}(\hat{x})<0$. Let $d\neq 0$ such that $d^{\top}A(\gamma_{-})d=0$ and define $e\coloneqq 2\left(\hat{x}^{\top}A(\gamma_{-})d+b(\gamma_{-})^{\top}d\right)$. For $\alpha\in{\mathbb{R}}$, let $(\hat{x}_{\alpha},\hat{t}_{\alpha})\coloneqq(\hat{x}+\alpha d,\hat{t}+\alpha e)$.

A short calculation shows that for any $\alpha\in{\mathbb{R}}$, we have

[TABLE]

Similarly, for any $\alpha\in{\mathbb{R}}$,

[TABLE]

As $d^{\top}A(\gamma_{-})d=0$ and $d\neq 0$, Assumption 1 implies that $d^{\top}A(\gamma_{+})d>0$. We see that $q(\gamma_{+},\hat{x}_{\alpha})-\hat{t}_{\alpha}$ is strongly convex in $\alpha$. As $q_{1}(\hat{x})<0$, we have $q(\gamma_{+},\hat{x})-\hat{t}<q(\gamma_{-},\hat{x})-\hat{t}$. Thus, there are values $\alpha_{1}<0<\alpha_{2}$ such that $q(\gamma_{+},\hat{x}_{\alpha_{i}})-\hat{t}_{\alpha_{i}}=q(\gamma_{-},\hat{x}_{\alpha_{i}})-\hat{t}_{\alpha_{i}}$ for $i=1,2$.

Noting that $\gamma_{-}\neq\gamma_{+}$ and $q(\gamma_{-},\hat{x}_{\alpha_{i}})=q(\gamma_{+},\hat{x}_{\alpha_{i}})$, we conclude that $q_{0}(\hat{x}_{\alpha_{i}})=q(\gamma_{i},\hat{x}_{\alpha_{i}})\leq\hat{t}_{\alpha_{i}}$ and $q_{1}(\hat{x}_{\alpha_{i}})=0$. Thus, $(\hat{x}_{\alpha_{i}},\hat{t}_{\alpha_{i}})\in\mathfrak{S}$ for $i=1,2$. We conclude $(\hat{x},\hat{t})\in\operatorname{conv}(\mathfrak{S})$.∎

Remark 4**.**

The proof of Lemma 3 suggests a simple rounding scheme from the convex relaxation to the original nonconvex problem: given $\hat{x}\in{\mathbb{R}}^{n}$, let $d$ be an eigenvector of eigenvalue zero for either $A(\gamma_{\pm})$ (depending on the sign of $q_{1}(\hat{x})$) and move $\alpha\geq 0$ units in the direction of either $\pm d$ (depending on the sign of $e$ defined in the proof) until $q_{1}(\hat{x}\pm\alpha d)=0$. This rounding scheme guarantees that $q_{0}(\hat{x}\pm\alpha d)\leq\max\left\{q(\gamma_{-},\hat{x}),q(\gamma_{+},\hat{x})\right\}$.

We are now ready to prove Theorem 1.

Proof of Theorem 1.

Lemmas 2 and 3 together imply

[TABLE]

Hence, we deduce that equality holds throughout the chain of inclusions.

In particular, the GTRS (4) can be rewritten

[TABLE]

It remains to prove that the minimum is achieved in each of the formulations of the GTRS above. It suffices to show that the minimum is achieved in the last formulation. Note $q(\gamma_{-},x)$ and $q(\gamma_{+},x)$ are both continuous functions of $x$, hence $\max\left\{q(\gamma_{-},x),q(\gamma_{+},x)\right\}$ is continuous. Next, taking $u\coloneqq\max\left\{c(\gamma_{-}),c(\gamma_{+})\right\}$ we have that $u$ is an upper bound on the optimal value. Moreover, because $\gamma^{*}\in(\gamma_{-},\gamma_{+})$, we can lower bound $\max\left\{q(\gamma_{-},x),q(\gamma_{+},x)\right\}$, by $q(\gamma^{*},x)$. Consequently, it suffices to replace the feasible domain ${\mathbb{R}}^{n}$ in the last formulation with the set

[TABLE]

This set is bounded as $A(\gamma^{*})\succ 0$ and it is closed as it is the inverse image of $(-\infty,u]$ under a continuous map. Recalling that a continuous function on a compact set achieves its minimum concludes the proof. ∎

We next provide a numerical example illustrating Theorem 1.

Example 1**.**

Define the homogeneous quadratic functions $q_{i}(x)\coloneqq x^{\top}A_{i}x$ for $i=0,1$, where

[TABLE]

As $\det(A_{0})=-3$ and $\det(A_{1})=-1$, the matrices $A_{0}$ and $A_{1}$ must both have negative eigenvalues. Furthermore,

[TABLE]

Thus, Assumption 1 is satisfied.

We now compute $\gamma_{-}$ and $\gamma_{+}$. Note that as $A(\gamma)$ is a $2\times 2$ matrix, $A(\gamma)\succeq 0$ if and only if $\operatorname{tr}(A(\gamma))\geq 0$ and $\det(A(\gamma))\geq 0$. Note that $\operatorname{tr}(A(\gamma))=2\geq 0$ is satisfied for all $\gamma$. We compute

[TABLE]

This quantity is nonnegative if and only if $\left\lvert 2-\gamma\right\rvert\leq 1$. Thus $\gamma_{-}=1$ and $\gamma_{+}=3$. Theorem 1 then implies

[TABLE]

We plot the corresponding sets ${\cal S}$ and ${\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$ in Figure 1.

2.2 Proof of Theorem 2

Next, we prove Theorem 2 using a limiting argument and reducing it to Theorem 1.

Lemma 4**.**

Suppose $\Gamma$ is nonempty and write $\Gamma=[\gamma_{-},\gamma_{+}]$. Then, ${\operatorname{\overline{conv}}}({\cal S})\subseteq{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$.

Proof.

Note that ${\cal S}=\bigcap_{\gamma\geq 0}{\cal S}(\gamma)\subseteq{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. Containment then follows by taking the closed convex hull of both sides and noting that both ${\cal S}(\gamma_{-})$ and ${\cal S}(\gamma_{+})$ are closed and convex by Lemma 1. ∎

Lemma 5**.**

Under Assumption 2, we have that ${\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})\subseteq{\operatorname{\overline{conv}}}(\mathfrak{S})$.

Proof.

Let $(\hat{x},\hat{t})\in{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. It suffices to show that $(\hat{x},\hat{t}+\epsilon)\in\operatorname{conv}(\mathfrak{S})$ for all $\epsilon>0$.

We will perturb $A_{0}$ slightly to create a new GTRS instance. Let $\delta>0$ to be picked later. Define $A_{0}^{\prime}=A_{0}+\delta I_{n}$ and let all remaining data be unchanged, i.e.,

[TABLE]

We will denote all quantities related to the perturbed system with an apostrophe.

We claim that it suffices to show that there exists a $\delta>0$ small enough such that the GTRS defined by $q_{0}^{\prime}$ and $q_{1}^{\prime}$ satisfies Assumption 1 and $(\hat{x},\hat{t}+\epsilon)\in{\cal S}^{\prime}(\gamma_{-}^{\prime})\cap{\cal S}^{\prime}(\gamma_{+}^{\prime})$ . Indeed, suppose this is the case. Note that for any $x\in{\mathbb{R}}^{n}$, we have $q_{1}(x)=q^{\prime}_{1}(x)$ and $q_{0}(x)\leq q^{\prime}_{0}(x)$. Hence, $\mathfrak{S}^{\prime}\subseteq\mathfrak{S}$ and $\operatorname{conv}(\mathfrak{S}^{\prime})\subseteq\operatorname{conv}(\mathfrak{S})$. Then applying Theorem 1 gives $(\hat{x},\hat{t}+\epsilon)\in{\cal S}^{\prime}(\gamma_{-}^{\prime})\cap{\cal S}^{\prime}(\gamma_{+}^{\prime})=\operatorname{conv}(\mathfrak{S}^{\prime})\subseteq\operatorname{conv}(\mathfrak{S})$ as desired.

We pick $\delta>0$ small enough such that

[TABLE]

This is possible as the expression on the left of each inequality is continuous in $\delta$ and is strictly satisfied if $\delta=0$. Then, noting that $A^{\prime}(\gamma^{*})=A(\gamma^{*})+\delta I_{n}\succ 0$, we have that the GTRS defined by $q_{0}^{\prime}$ and $q_{1}^{\prime}$ satisfies Assumption 1.

It remains to show that $q^{\prime}(\gamma^{\prime}_{+},\hat{x})\leq(\hat{t}+\epsilon)$ and $q^{\prime}(\gamma^{\prime}_{-},\hat{x})\leq(\hat{t}+\epsilon)$. We compute

[TABLE]

The first inequality follows by noting $q^{\prime}(\gamma,x)=q(\gamma,x)+\delta\left\lVert x\right\rVert^{2}$, the second inequality follows from our assumptions on $\delta$, and the third line follows from the assumption that $(\hat{x},\hat{t})\in{\cal S}(\gamma_{+})$. A similar calculation shows $q^{\prime}(\gamma^{\prime}_{-},\hat{x})\leq(t+\epsilon)$. This concludes the proof. ∎

We are now ready to prove Theorem 2.

Proof of Theorem 2.

Lemmas 4 and 5 together imply

[TABLE]

Hence, we deduce that equality holds throughout the chain of inclusions.

In particular, the GTRS (4) can be rewritten

[TABLE]

2.3 Removing the nonconvex assumptions

As part of our Assumptions 1 and 2, we assume that $A_{0}$ and $A_{1}$ both have negative eigenvalues, i.e., that both $q_{0}$ and $q_{1}$ are nonconvex. These assumptions are made for ease of presentation and to highlight the novel contributions of this work. Indeed, the proofs of Theorems 1 and 2 can be modified to additionally cover all four cases of convex/nonconvex objective and constraint functions. We remark that the resulting theorem statement for the case of a nonconvex objective function and a strongly convex constraint function coincides with that of Ho-Nguyen and Kılınç-Karzan [15].

In this section we record more general versions Theorems 1 and 2. Their proofs are completely analogous to the original proofs and are deferred to Appendix A.

{restatable}

theorempdGeneralConvHull

Suppose there exists $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succ 0$. Consider the closed nonempty interval $\Gamma\coloneqq\left\{\gamma\in{\mathbb{R}}_{+}:\,A(\gamma)\succeq 0\right\}$. Let $\gamma_{-}$ denote its leftmost endpoint.

• •

If $\Gamma$ is bounded above, let $\gamma_{+}$ denote its rightmost endpoint. Then,

[TABLE]

In particular, we have $\min_{x\in{\mathbb{R}}^{n}}\left\{q_{0}(x):\,q_{1}(x)\leq 0\right\}=\min_{x\in{\mathbb{R}}^{n}}\max\left\{q(\gamma_{-},x),\,q(\gamma_{+},x)\right\}$.

• •

If $\Gamma$ is not bounded above, then $q_{1}(x)$ is convex and

[TABLE]

In particular, we have $\min_{x\in{\mathbb{R}}^{n}}\left\{q_{0}(x):\,q_{1}(x)\leq 0\right\}=\min_{x\in{\mathbb{R}}^{n}}\left\{q(\gamma_{-},x):\,q_{1}(x)\leq 0\right\}$.

{restatable}

theorempsdGeneralConvHull

Suppose there exists $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succeq 0$. Consider the closed nonempty interval $\Gamma\coloneqq\left\{\gamma\in{\mathbb{R}}_{+}:\,A(\gamma)\succeq 0\right\}$. Let $\gamma_{-}$ denote its leftmost endpoint.

• •

If $\Gamma$ is bounded above, let $\gamma_{+}$ denote its rightmost endpoint. Then,

[TABLE]

In particular, $\inf_{x\in{\mathbb{R}}^{n}}\left\{q_{0}(x):\,q_{1}(x)\leq 0\right\}=\inf_{x\in{\mathbb{R}}^{n}}\max\left\{q(\gamma_{-},x),\,q(\gamma_{+},x)\right\}$.

• •

If $\Gamma$ is not bounded above, then $q_{1}(x)$ is convex and

[TABLE]

In particular, $\inf_{x\in{\mathbb{R}}^{n}}\left\{q_{0}(x):\,q_{1}(x)\leq 0\right\}=\inf_{x\in{\mathbb{R}}^{n}}\left\{q(\gamma_{-},x):\,q_{1}(x)\leq 0\right\}$.

These results admit further nontrivial generalizations involving multiple quadratics; we refer the interested readers to our follow up work [34].

Remark 5**.**

Yıldıran [37]** proves a convex hull result for a set defined by two strict quadratic constraints. Modaresi and Vielma [24] then show that given a particular topological assumption, that the appropriate closed versions of Yıldıran [37]’s results also hold. We discuss these results in the context of the convex hull results we have presented thus far. Given $q_{0}$ and $q_{1}$ we will consider the quadratic functions $q_{0}(x)-t$ and $q_{1}(x)$ in the variables $(x,t)$. As [37] works with homogeneous quadratics, we introduce an extra variable to get homogeneous quadratic forms. Define

[TABLE]

Yıldıran [37]** uses the aggregation weights $\gamma$ where $Q(\gamma)$ has exactly one negative eigenvalue. Note that for all $\gamma\geq 0$, the lower right $2\times 2$ block of $Q(\gamma)$ is invertible. Thus, we may take the Schur complement of this block in $Q(\gamma)$:

[TABLE]

Recall that Schur complements preserve inertia. In other words, $Q(\gamma)$ and

[TABLE]

have the same number of negative eigenvalues. Noting that the lower right $2\times 2$ block has exactly one negative eigenvalue, we conclude that $Q(\gamma)$ has exactly one negative eigenvalue if and only if $A(\gamma)\succeq 0$. The result presented by Yıldıran [37] then implies

[TABLE]

when $\gamma_{+}$ exists and

[TABLE]

otherwise.

One can then verify the topological assumption of Modaresi and Vielma [24], namely that ${\cal S}\subseteq\overline{\operatorname{int}({\cal S})}$ the closure of the interior of ${\cal S}$. Thus, combining these two results gives an alternate proof of Theorems 2.3 and 2.3.

We believe our analysis is simpler and more direct. In particular, our analysis takes advantage of the epigraph structure present in our sets and immediately implies a rounding procedure via Lemma 3. In addition, our results are more refined when Assumption 1 or 2 hold as we can also characterize the (closed) convex hull of the set $\mathfrak{S}$ and show that it is equal to that of ${\cal S}$. This particular distinction between $\mathfrak{S}$ and ${\cal S}$ has a number of interesting implications in equality-, interval-, or hollow-constrained GTRS, and we discuss these results in the following section.

3 Nonintersecting constraints

There have been a number of works considering interval-, equality-, or hollow-constrained variants of the GTRS [3, 2, 19, 17, 30, 32, 25, 33, 36] (see [15, Section 3.3] and references therein for extensions of the TRS and their applications). In this section, we extend our (closed) convex hull results in the presence of a general nonintersecting constraint. This allows us to handle multiple variants of the GTRS simultaneously.

Specifically, we will impose an additional requirement $x\in\Omega$. The new form of the GTRS will be

[TABLE]

Let ${\cal S}_{\Omega}$ denote the set of feasible points $(x,t)$, i.e.,

[TABLE]

We will assume that $\Omega\subseteq{\mathbb{R}}^{n}$ satisfies the following nonintersecting condition.

Assumption 3**.**

The set $\Omega\subseteq{\mathbb{R}}^{n}$ satisfies $\left\{x\in{\mathbb{R}}^{n}:\leavevmode\nobreak\ q_{1}(x)=0\right\}\subseteq\Omega$.

The following two corollaries to Theorems 1 and 2 follow immediately by noting that $\mathfrak{S}\subseteq{\cal S}_{\Omega}\subseteq{\cal S}$ holds under Assumption 3.

Corollary 1**.**

Suppose Assumptions 1 and 3 hold. Then,

[TABLE]

Proof.

Under Assumptions 1 and 3, we get the following chain of inclusions

[TABLE]

where the first subset relation follows ${\cal S}_{\Omega}\subseteq{\cal S}$ (by definition of the set ${\cal S}_{\Omega}$), the equality relation follows from Theorem 1, and the last subset relation follows from $\mathfrak{S}\subseteq{\cal S}_{\Omega}$ (by Assumption 3). We conclude that $\operatorname{conv}({\cal S}_{\Omega})=\operatorname{conv}({\cal S})$. By Theorem 1, we know that $\operatorname{conv}({\cal S})={\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. ∎

Corollary 2**.**

Suppose Assumptions 2 and 3 hold. Then,

[TABLE]

Proof.

Applying Assumptions 2 and 3 and Theorem 2, we get the following chain of inclusions

[TABLE]

We conclude that $\overline{\operatorname{conv}}({\cal S}_{\Omega})=\overline{\operatorname{conv}}({\cal S})$. By Theorem 2, we know that $\overline{\operatorname{conv}}({\cal S})={\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. ∎

Remark 6**.**

These two corollaries show that nonintersecting constraints in the GTRS may be ignored. Consider for example the interval-constrained GTRS. Define

[TABLE]

Then, clearly Assumption 3 is satisfied. Under Assumption 2, we have

[TABLE]

Thus, the value of the interval-constrained GTRS is the same as the GTRS under Assumption 2. Similarly, the $\Omega$ sets arising from equality- or hollow-constrained GTRS also satisfy Assumption 3. Hence, under Assumption 2, the additional constraints in these variants of the GTRS can also be dropped.

4 Solving the convex reformulation in linear time

In this section we present algorithms, inspired by Theorem 1, for approximately solving the GTRS. Note that Theorem 1 gives a tight convex reformulation of the GTRS: under Assumption 1,

[TABLE]

Then given a solution to the convex reformulation on the right, Lemma 3 gives a rounding scheme to recover a solution to the original GTRS on the left.

In order to establish an explicit running time of an algorithm based on the above idea, we must carefully handle a number of numerical issues. In practice, we cannot expect to compute $\gamma_{\pm}$ exactly. Instead, we will show how to compute estimates $\tilde{\gamma}_{\pm}$ of $\gamma_{\pm}$ up to some accuracy $\delta$. We will take care to pick $\tilde{\gamma}_{\pm}$ satisfying the relation $[\tilde{\gamma}_{-},\tilde{\gamma}_{+}]\subseteq[\gamma_{-},\gamma_{+}]$ so that the quadratic forms defined by $A(\tilde{\gamma}_{-})$ and $A(\tilde{\gamma}_{+})$ are convex. Based on the estimates $\tilde{\gamma}_{\pm}$, we will then formulate and solve the convex optimization problem

[TABLE]

Finally, given an (approximate) solution to the convex problem $\widetilde{\operatorname{Opt}}$, Lemma 3 tells us how to construct a solution to the original nonconvex GTRS using specific eigenvectors. Again, we will need to handle numerical issues that arise from not being able to compute these eigenvectors exactly.

Throughout this section, we will work under the following assumption.

Assumption 4**.**

• •

There exists some $\gamma^{*}\geq 0$ such that $A(\gamma^{*})\succ 0$,

• •

$\left\lVert A_{0}\right\rVert,\left\lVert A_{1}\right\rVert,\left\lVert b_{0}\right\rVert,\left\lVert b_{1}\right\rVert,\left\lvert c_{1}\right\rvert\leq 1$.

Remark 7**.**

Note that the first part of Assumption 4 is simply Assumption 1. We make this assumption so that we may use the convex reformulation guaranteed by Theorem 1. Assumption 1 is commonly used in GTRS algorithms; see e.g., Jiang and Li [18, Assumption 2.3] and the discussion following it. The second part of Assumption 4 can be achieved for an arbitrary pair $q_{0}$ and $q_{1}$ by simply scaling each quadratic by a positive scalar. Note that any optimal (respectively feasible) solution remains optimal (respectively feasible) when $q_{0}$ (respectively $q_{1}$) is scaled by a positive scalar.

We will analyze the running time of our algorithm in terms of $N$, the number of nonzero entries in $A_{0}$ and $A_{1}$, $\epsilon$, the additive error, $p$, the failure probability, and $n$, the dimension. In addition, the running time of our algorithm depends on certain regularity parameters of the pair $q_{0}$ and $q_{1}$ defined below.

Definition 1**.**

Let $q_{0}$, $q_{1}$ satisfy Assumption 4. Define

[TABLE]

We say that $q_{0}$ and $q_{1}$ are $(\xi,\zeta)$ regular if $0<\xi\leq\xi^{*}$ and $\zeta\geq\zeta^{*}$. Define $\kappa^{*}=\zeta^{*}/\xi^{*}$. When $(\xi,\zeta)$ are clear from context we will write $\kappa\coloneqq\zeta/\xi$.

In our analysis, we will frequently use the inequalities $\kappa,\zeta,\xi^{-1}\geq 1$, which for example imply $\kappa^{2}\geq\kappa$ and $1+\kappa\leq 2\kappa$, and the inequalities $\gamma_{-}\leq\gamma_{+}\leq\zeta$, which for example under Assumption 4 imply $\left\lVert A(\gamma_{+})\right\rVert\leq 1+\zeta\leq 2\zeta$.

Remark 8**.**

Jiang and Li [18]** present a different linear-time algorithm for solving the GTRS. In their paper, they assume they are given a regularity parameter $\xi_{\text{JL}}$ as input. This parameter must satisfy $\xi_{\text{JL}}\leq\xi_{\text{JL}}^{*}$ where

[TABLE]

We now discuss how our regularity parameters, $\xi^{*}_{\text{us}}$, $\zeta^{*}$, and $\kappa^{*}\coloneqq\frac{\zeta^{*}}{\xi^{*}_{\text{us}}}$ relate to $\xi^{*}_{\text{JL}}$. For simplicity, we will assume

[TABLE]

We claim $\zeta^{*}\leq(-\lambda_{\min}(A_{1}))^{-1}$. Indeed, let $x$ be a unit eigenvector corresponding to $\lambda_{\min}(A_{1})$. Then, for any $\gamma>(-\lambda_{\min}(A_{1}))^{-1}$, we have

[TABLE]

The role played by the bound $\gamma_{+}\leq\zeta$ in our analysis is similar to the role of the bound $\xi_{\text{JL}}\leq-\lambda_{\min}(A_{1})$ in the analysis presented by Jiang and Li [18].

We claim that

[TABLE]

and that the lower bound is sharp. Indeed, by performing the transformation $\mu=\frac{1}{1+\gamma}$, we can rewrite

[TABLE]

which we can clearly bound above by $\xi^{*}_{\text{us}}$. On the other hand, noting that any optimizer, $\gamma$, of the above problem must lie in $[0,\gamma_{+}]=[0,\zeta^{*}]$, we can lower bound

[TABLE]

We now construct a simple example for which the lower bound, $\xi^{*}_{\text{JL}}\geq\tfrac{1}{2\kappa^{*}}$, is sharp. Let $\alpha>0$ and define

[TABLE]

It is simple to see that $\left\lVert A_{0}\right\rVert=\left\lVert A_{1}\right\rVert=1$, $\xi^{*}_{\text{us}}=\tfrac{\alpha}{2+\alpha}$ and $\zeta^{*}=1+\alpha$. In particular, $\kappa^{*}=\tfrac{2+3\alpha+\alpha^{2}}{\alpha}$. On the other hand, we can compute

[TABLE]

Then, letting $\alpha\to 0$, we have $\kappa^{*}=\tfrac{2+o(1)}{\alpha}$ and $\xi^{*}_{\text{JL}}=\tfrac{\alpha}{4+o(1)}$.

In view of the (closed) convex hull results presented in Theorems 1 and 2, we believe that the right notion of regularity should depend on the parameterization $A_{0}+\gamma A_{1}$ as opposed to $\mu A_{0}+(1-\mu)A_{1}$. We compare the running time of the algorithm presented by Jiang and Li [18] and the running time of our algorithms in Remark 12.

We will assume that we have access to these regularity parameters within our algorithms.

Assumption 5**.**

Assume we have algorithmic access to a pair $(\xi,\zeta)$ such that $q_{0}$ and $q_{1}$ are $(\xi,\zeta)$-regular and a $\hat{\gamma}$ satisfying $\lambda_{\min}(A(\hat{\gamma}))\geq\xi$.

Remark 9**.**

Assumption 5 is quite reasonable. Indeed, there are simple and efficient binary search schemes to find constant factor approximations of $\xi^{*}$ and $\zeta^{*}$ and a corresponding $\hat{\gamma}$. We detail one such algorithm in Appendix B. We remark that a similar assumption is made by Jiang and Li [18]: they assume they are given access to $\xi_{\text{JL}}$ and present an algorithm for computing a corresponding $\hat{\mu}$ (see Remark 8). Another algorithm for finding $\hat{\gamma}$ is presented by Guo et al. [12] in the language of matrix pencil definiteness.

We now fix the accuracy222Our definition of accuracy is presented in (11). to which we will compute our estimates $\tilde{\gamma}_{\pm}$. Define

[TABLE]

The framework for our approach is shown in Algorithm 1.

Note that by Definition 1, we have $\kappa^{2}\xi=\zeta^{2}/\xi\geq 1$. Thus the requirement $0<\epsilon\leq\kappa^{2}\xi$ in Algorithm 1 is not a practical issue: given $\epsilon>\kappa^{2}\xi$, we can simply run our algorithm with $\epsilon^{\prime}=1$ and return a solution with a better error guarantee.

This section is structured as follows. In Section 4.1, we prove that when $\delta$ is picked according to (10), $\widetilde{\operatorname{Opt}}$ is within $\epsilon/2$ of $\operatorname{Opt}$. In Section 4.2 we show how to compute $\tilde{\gamma}_{-}$ and $\tilde{\gamma}_{+}$ to satisfy (11). Then in Section 4.3, we present an algorithm due to Nesterov [27] and show that it can be used to efficiently solve for $\widetilde{\operatorname{Opt}}$ up to accuracy $\epsilon/2$. At the end of Section 4.3, we present Theorem 4, which collects the results of the previous subsections and formally analyzes the runtime of Algorithm 1. In Section 4.4, we give a rounding scheme for finding a solution to the original GTRS (1) given a solution to the convex reformulation. Finally, in Section 4.5, we show that the running times of our algorithms can be significantly improved in situations where it is easy to compute $\gamma_{\pm}$ and zero eigenvectors of $A(\gamma_{\pm})$.

4.1 Perturbation analysis of the convex reformulation

In this subsection, we show that the perturbed convex reformulation, $\widetilde{\operatorname{Opt}}$, approximates the true convex reformulation, $\operatorname{Opt}$, up to an additive error of $\epsilon/2$ when $\delta$ is picked as in (10). We will assume that step 2 of Algorithm 1 is successful, i.e., we have $\tilde{\gamma}_{\pm}$ satisfying (11).

Recall the definition of $\delta$ in (10). As we require $\epsilon\leq\kappa^{2}\xi$, we will have

[TABLE]

It is easy to see that $\lambda_{\min}(A(\gamma))$ is a $1$-Lipschitz function in $\gamma$. Then recalling that $\lambda_{\min}(A(\gamma_{\pm}))=0$ and $\lambda_{\min}(A(\hat{\gamma}))\geq\xi$, we deduce the containment $\hat{\gamma}\in(\gamma_{-}+\delta,\gamma_{+}-\delta)$. This, along with (11), implies

[TABLE]

Recall the perturbed reformulation

[TABLE]

For notational convenience, let $f(x)\coloneqq\max\left\{q(\gamma_{-},x),q(\gamma_{+},x)\right\}$ and let $\tilde{f}(x)\coloneqq\max\left\{q(\tilde{\gamma}_{-},x),q(\tilde{\gamma}_{+},x)\right\}$. Let $x^{*}$ and $\tilde{x}^{*}$ denote optimizers of $\operatorname{Opt}$ and $\widetilde{\operatorname{Opt}}$ respectively.

Lemma 6**.**

For any fixed $x\in{\mathbb{R}}^{n}$, we have $\tilde{f}(x)\leq f(x)$. In particular, $\widetilde{\operatorname{Opt}}\leq\operatorname{Opt}$.

Proof.

Note that $q(\gamma,x)$ is a linear function in $\gamma$. Hence, for any fixed $x\in{\mathbb{R}}^{n}$, the containment $[\tilde{\gamma}_{-},\tilde{\gamma}_{+}]\subseteq[\gamma_{-},\gamma_{+}]$ implies $\tilde{f}(x)\leq f(x)$. We deduce

[TABLE]

To show $\widetilde{\operatorname{Opt}}\geq\operatorname{Opt}-\epsilon/2$, we will show that $x^{*}$ and $\tilde{x}^{*}$ lie in a ball of bounded radius and that $\tilde{f}$ approximates $f$ uniformly on this ball.

Lemma 7**.**

Let $x^{*}$ and $\tilde{x}^{*}$ be optimizers of $\operatorname{Opt}$ and $\widetilde{\operatorname{Opt}}$ respectively. Then $x^{*},\tilde{x}^{*}\in B(0,5\kappa)$.

Proof.

By picking the feasible solution $0\in{\mathbb{R}}^{n}$ and Lemma 6, we have a trivial upper bound on $\widetilde{\operatorname{Opt}}$ and $\operatorname{Opt}$:

[TABLE]

By the first part of (12), we have

[TABLE]

where the last inequality follows from the assumption that $\lambda_{\min}(A(\hat{\gamma}))\geq\xi$. Then,

[TABLE]

The last relation holds since $\max\left\{c(\gamma_{-})-c(\hat{\gamma}),c(\gamma_{+})-c(\hat{\gamma})\right\}=\max\{(\gamma_{-}-\hat{\gamma})c_{1},(\gamma_{+}-\hat{\gamma})c_{1}\}\leq|c_{1}|\gamma_{+}\leq\zeta$. Then, by completing the square

[TABLE]

where in the third line, we used Assumption 4 and the bound $\left\lVert b(\hat{\gamma})\right\rVert\leq\left\lVert b_{0}\right\rVert+\gamma_{+}\left\lVert b_{1}\right\rVert\leq 2\zeta$. ∎

Lemma 8**.**

If $\left\lVert\hat{x}\right\rVert\leq 5\kappa$, then $\tilde{f}(\hat{x})\geq f(\hat{x})-\epsilon/2$. In particular, $\widetilde{\operatorname{Opt}}\geq\operatorname{Opt}-\epsilon/2$,

Proof.

Recall that $\delta\coloneqq\frac{\epsilon}{72\kappa^{2}}$. Let $\hat{x}\in{\mathbb{R}}^{n}$ such that $\left\lVert\hat{x}\right\rVert\leq 5\kappa$. We compute

[TABLE]

where the first inequality follows from (12), the second inequality follows from Assumption 4, and the third inequality follows from the bound $\left\lVert\hat{x}\right\rVert\leq 5\kappa$. ∎

4.2 Approximating $\gamma_{-}$ and $\gamma_{+}$

In this subsection, we show how to approximate $\gamma_{-}$ and $\gamma_{+}$ and provide an explicit running time analysis of this procedure. Our developments rely on the fact that $\lambda_{\min}(A(\gamma))$ is a concave function in $\gamma$ and that $\gamma_{-}$ and $\gamma_{+}$ are the unique zeros of this function.

Lemma 9**.**

$\lambda_{\min}(A(\gamma))$* is a concave function in $\gamma$.*

Proof.

By Courant-Fischer Theorem, $\lambda_{\min}(A(\gamma))=\min_{\left\lVert x\right\rVert=1}x^{\top}A(\gamma)x$. Note that for any fixed $x\in{\mathbb{R}}^{n}$, the expression $x^{\top}A(\gamma)x$ is linear in $\gamma$. Then, the result follows upon recalling that the minimum of concave (in our case linear) functions is concave. ∎

Let us also state a simple property of the function $\lambda_{\min}(A(\gamma))$.

Lemma 10**.**

1. (i)

Suppose $\gamma\leq\hat{\gamma}$, then $\left\lvert\gamma-\gamma_{-}\right\rvert\leq\kappa\left\lvert\lambda_{\min}(A(\gamma))\right\rvert.$ 2. (ii)

Suppose $\gamma\geq\hat{\gamma}$, then $\left\lvert\gamma-\gamma_{+}\right\rvert\leq\kappa\left\lvert\lambda_{\min}(A(\gamma))\right\rvert.$

Proof.

We only prove the first statement as the second statement follows similarly. Let $\gamma\leq\hat{\gamma}$. From the concavity of $\lambda_{\min}(A(\gamma))$, we have

[TABLE]

where in the second inequality we used the definition of $\xi$ in Definition 1 and the bound $\hat{\gamma}-\gamma_{-}\leq\gamma_{+}\leq\zeta$. Noting $\zeta/\xi=\kappa$ and rearranging terms completes the proof. ∎

We will use the Lanczos method for approximating the most negative eigenvalue (and a corresponding eigenvector) of a sparse matrix. This algorithm, along with Lemma 10, will allow us to binary search over the range $[0,\zeta]$ for the zeros of the function $\lambda_{\min}(A(\gamma))$.

Lemma 11** ([21]).**

There exists an algorithm, $\text{ApproxEig}(A,\rho,\eta,p_{\text{eig}})$, which given a symmetric matrix $A\in{\mathbb{R}}^{n\times n}$, $\rho$ such that $\left\lVert A\right\rVert\leq\rho$, and parameters $\eta,p_{\text{eig}}>0$, will, with probability at least $1-p_{\text{eig}}$, return a unit vector $x\in{\mathbb{R}}^{n}$ such that $x^{\top}Ax\leq\lambda_{\min}(A)+\eta$. This algorithm runs in time

[TABLE]

where $N$ is the number of nonzero entries in $A$.

Consider ApproxGammaPlus (Algorithm 2) for computing $\tilde{\gamma}_{+}$ up to accuracy $\delta$. A similar algorithm can be used to compute $\tilde{\gamma}_{-}$ up to accuracy $\delta$ and is omitted.

{restatable}

lemmaApproxGammaPlusLemma

Given $q_{0}$, $q_{1}$ satisfying Assumption 4, $(\xi,\zeta)$ and $\hat{\gamma}$ satisfying Assumption 5, $\delta>0$, and $p_{\tilde{\gamma}_{+}}$, ApproxGammaPlus (Algorithm 2) outputs $\tilde{\gamma}_{+}$ satisfying

[TABLE]

with probability $1-p_{\tilde{\gamma}_{+}}$. This algorithm runs in time

[TABLE]

Proof.

We condition on the event that ApproxEig succeeds every time it is called. By the union bound, this happens with probability at least $1-p_{\tilde{\gamma}_{+}}$.

Suppose the algorithm outputs at step 3.(e). Let $\gamma$ be the value of $\gamma$ on the round in which the algorithm stops, and $x$ the vector returned by ApproxEig in the corresponding iteration. Then, the stopping rule guarantees $x^{\top}A(\gamma)x\in[\delta/4\kappa,\delta/\kappa]$. As we have conditioned on ApproxEig succeeding, we deduce

[TABLE]

In particular, $\lambda_{\min}(A(\gamma))\in[0,\delta/\kappa]$ and $\gamma\leq\gamma_{+}$. Applying Lemma 10 gives

[TABLE]

We conclude $\gamma_{+}-\delta\leq\gamma\leq\gamma_{+}$.

We now show that this algorithm outputs within $T$ rounds. Let

[TABLE]

Recalling that $\lambda_{\min}(A(\gamma))$ is $1$-Lipschitz in $\gamma$, we deduce that $\left\lvert P\right\rvert\geq\delta/2\kappa$. Note also that $\lambda_{\min}(A(\hat{\gamma}))\geq\xi\geq\delta\geq 3\delta/4\kappa$ thus $P$ is a connected interval.

Suppose for the sake of contradiction that the algorithm fails to output in each of the $T$ rounds. Note that $P\subseteq[s_{0},t_{0}]$. We will show by induction that $P\subseteq[s_{k},t_{k}]$ for every $k\in\left\{1,\dots,T\right\}$. Let $k\in\left\{0,\dots,T-1\right\}$. By assumption, the algorithm fails to output in round $k$. This can happen in two ways: If $x^{\top}A(\gamma)x<\delta/4\kappa$, then $x$ certifies that $\gamma\notin P$ and $P\subseteq[s_{k},\gamma]$. If $x^{\top}A(\gamma)x>\delta/\kappa$, then as we have conditioned on ApproxEig succeeding, $\lambda_{\min}(A(\gamma))\geq\delta/\kappa-\delta/4\kappa$ and $P\subseteq[\gamma,t_{k}]$. In either case, we have that $P\subseteq[s_{k+1},t_{k+1}]$.

We conclude that $P$, an interval of length at least $\delta/2\kappa$, is contained in $[s_{T},t_{T}]$, an interval of length

[TABLE]

a contradiction. Thus, the algorithm outputs within $T$ rounds.

The running time of this algorithm follows from Lemma 11. ∎

Remark 10**.**

Similar algorithms for approximating $\gamma_{\pm}$ given $\hat{\gamma}$ have been proposed in the literature [25, 30, 1, 17]. However to our knowledge, this is the first analysis to establish an explicit convergence rate; see the discussion after Remark 2.11 in [17] on this issue.

4.3 Minimizing the maximum of two quadratic functions

In this subsection, we will assume that Algorithm 1 has successfully found $\tilde{\gamma}_{\pm}$ satisfying (11) and show how to approximately solve

[TABLE]

For the sake of readability, we will use the following notation in this subsection.

[TABLE]

In particular we have $\tilde{f}(x)=\max\left\{\tilde{f}_{0}(x),\tilde{f}_{1}(x)\right\}$.

Our analysis is based on Nesterov [27, Section 2.3.3], which proposes a high level algorithm for minimizing general minimax problems with smooth components. We state this algorithm (Algorithm 3) and its corresponding convergence rate in our context.

Theorem 3** ([27, Theorem 2.3.5]).**

Let $\tilde{f}_{0},\tilde{f}_{1}$ be $2L$-smooth333Recall that a convex quadratic function $x^{\top}Ax+2b^{\top}x+c$ is $2L$-smooth if and only if $A\preceq LI$. differentiable convex functions such that $\tilde{f}$ is bounded below. Let $\tilde{x}^{*}$ be an optimizer of $\tilde{f}$. Then the iterates $x_{k}$ produced by Algorithm 3 satisfy

[TABLE]

Lemma 12**.**

Let $x\in{\mathbb{R}}^{n}$. Then for $i=0,1$, we have

[TABLE]

Proof.

For $i=0,1$, we have

[TABLE]

where the second inequality follows from Assumption 4. ∎

Corollary 3**.**

Let $\tilde{f}_{0}$ and $\tilde{f}_{1}$ be the functions defined in (14). Let $\tilde{x}^{*}$ be an optimizer of $\tilde{f}$. Then the iterates $x_{k}$ produced by Algorithm 3 satisfy

[TABLE]

In particular, after $k=O\left(\kappa\sqrt{\zeta/\epsilon}\right)$ iterations, the solution $x_{k}$ satisfies $\tilde{f}(x_{k})-\tilde{f}(\tilde{x}^{*})\leq\epsilon/2$.

Proof.

We have that $\tilde{f}_{0}$ and $\tilde{f}_{1}$ are both $2(2\zeta)$-smooth by Assumption 4 and Definition 1. Moreover, $\tilde{f}(x)\geq q(\hat{\gamma},x)$ is bounded below. Thus, we may apply Theorem 3.

We bound the initial primal gap as follows:

[TABLE]

where the third line follows from definition (see (14)), the fourth line follows from the ordering $\tilde{\gamma}_{-}\leq\tilde{\gamma}_{+}\leq\zeta$, the fifth line follows from Lemmas 7 and 12, and the last line follows from the trivial bounds $\kappa\geq 1$ and $\zeta\geq 1$.

Using Lemma 7 again, we also have $\frac{L}{2}\left\lVert\tilde{x}^{*}\right\rVert^{2}=\frac{2\zeta}{2}\left\lVert\tilde{x}^{*}\right\rVert^{2}\leq 25\kappa^{2}\zeta$. The result follows by combining these bounds. ∎

It remains to analyze the runtime of each iteration. Aside from computation of $x_{k+1}$, it is clear that the quantities in each iteration can be computed in $O(N)$ time. Below, we derive a closed form expression for $x_{k+1}$ where each of the quantities can be computed in $O(N)$ time.

Lemma 13**.**

For any $y\in{\mathbb{R}}^{n}$, the quantity

[TABLE]

can be computed in $O(N)$ time.

Proof.

Fix $y\in{\mathbb{R}}^{n}$. We begin by recentering the quadratic functions in the objective.

[TABLE]

Here, $z_{i}$ and $h_{i}$ are defined to be the square-bracketed terms from the preceding line. It is clear that the minimizing $x$ must belong to the line segment $[z_{0},z_{1}]$. We will parameterize $x=z_{0}+\alpha(z_{1}-z_{0})$ where $\alpha\in[0,1]$.

[TABLE]

We solve for $\alpha$ by setting the two terms inside the maximum equal. A simple calculation yields that the two quadratics are equal when

[TABLE]

If $\bar{\alpha}$ is between $[0,1]$, let $\alpha^{*}=\bar{\alpha}$. Else let $\alpha^{*}=0$ (respectively $\alpha^{*}=1$) when $\bar{\alpha}<0$ (respectively $\bar{\alpha}>1$).

Then,

[TABLE]

Each of the quantities on the right hand side (namely $\alpha^{*}$, $z_{i}$) can be computed in $O(N)$ time. ∎

Combining Corollary 3 and Lemma 13 gives the following corollary.

Corollary 4**.**

Let $\tilde{f}_{0},\tilde{f}_{1}$ be the functions defined in (14). There exists an algorithm which outputs $\tilde{x}$ satisfying $\tilde{f}(\tilde{x})\leq\widetilde{\operatorname{Opt}}+\epsilon/2$ running in time

[TABLE]

Remark 11**.**

Jiang and Li [17]** present a saddle-point-based first-oder algorithm for approximating $\widetilde{\operatorname{Opt}}$. By instantiating their algorithm with the initial iterate $x_{0}=0$ and applying our Lemma 7 to bound $\left\lVert x_{0}-\tilde{x}^{*}\right\rVert^{2}$, we have that [17, Algorithm 1] produces an $\epsilon/2$-optimal solution to the convex reformulation in time

[TABLE]

Therefore, the dependences on $\epsilon$, $\kappa$, and $\zeta$ of this algorithm are worse than that of the algorithm described in Corollary 4. Note that [17] does not present an analysis of the complexity of finding the approximate generalized eigenvalue $\tilde{\gamma}_{\pm}$ (needed to construct $\widetilde{\operatorname{Opt}}$) or how $\widetilde{\operatorname{Opt}}$ relates to $\operatorname{Opt}$.

By combining Lemmas 6, 8, and 2 and Corollary 4, we arrive at the following main theorem on the overall computational complexity of our approach.

Theorem 4**.**

Given $q_{0},q_{1}$ satisfying Assumption 4, $(\xi,\zeta)$ and $\hat{\gamma}$ satisfying Assumption 5, error parameter $0<\epsilon\leq\kappa^{2}\xi$, and failure probability $p>0$, ApproxConvex (Algorithm 1) outputs $\tilde{\gamma}_{-}$, $\tilde{\gamma}_{+}$ and $\tilde{x}\in{\mathbb{R}}^{n}$ such that

[TABLE]

with probability $1-p$. This algorithm runs in time

[TABLE]

4.4 Finding an approximate optimizer of the GTRS

Let $\tilde{x}\in{\mathbb{R}}^{n}$ be the approximate optimizer output by Algorithm 1. In this subsection, we show how to use $\tilde{x}$ to construct an $\bar{x}$ approximately minimizing the original GTRS (1). Our algorithm will follow the proof of Theorem 1 (in particular Lemma 3).

We present our algorithm, ApproxGTRS, as Algorithm 4. ApproxGTRS will use ApproxConvex as a subroutine. Given an additive error $\epsilon_{\text{round}}$, ApproxGTRS will call ApproxConvex with additive error $\epsilon_{\text{convex}}$. We will write these parameters as $\epsilon_{r}$ and $\epsilon_{c}$ for short.

Note that by Definition 1, we have $\kappa^{3}\xi\geq 1$. Thus, as before, the requirement $0<\epsilon_{r}\leq\kappa^{3}\xi$ in Algorithm 4 is not a practical issue: given $\epsilon_{r}>\kappa^{3}\xi$, we can simply run our algorithm with $\epsilon_{r}^{\prime}=\kappa^{3}/\xi$ and return a solution with a better error guarantee.

The next lemma bounds $\left\lVert\tilde{x}\right\rVert$. Its proof follows the proof of Lemma 7 with minor adjustments (in particular, the upper bound of (13) is replaced with $\tilde{f}(\tilde{x})\leq\max\left\{c(\gamma_{-}),c(\gamma_{+})\right\}+\epsilon/2$; see Corollary 4) and is omitted.

Lemma 14**.**

Let $\tilde{x}\in{\mathbb{R}}^{n}$ satisfy $\tilde{f}(\tilde{x})\leq\tilde{\operatorname{Opt}}+\epsilon/2$. Then $\tilde{x}\in B(0,6\kappa)$.

We are now ready to prove a formal guarantee on Algorithm 4.

Theorem 5**.**

Given $q_{0},q_{1}$ satisfying Assumption 4, $(\xi,\zeta)$ and $\hat{\gamma}$ satisfying Assumption 5, error parameter $0<\epsilon_{r}\leq\kappa^{3}\xi$, and failure probability $p_{r}$, ApproxGTRS (Algorithm 4) outputs $\bar{x}$ such that

[TABLE]

with probability $1-p_{r}$. This algorithm runs in time

[TABLE]

Proof.

We condition on the event that Algorithm 1 succeeds and the ApproxEig call in step 4.(a) or 5.(a) succeeds. By the union bound, this happens with probability at least $1-p_{r}$. As in Lemma 3, we will split the analysis into three cases: (i) $q_{1}(\tilde{x})=0$, (ii) $q_{1}(\tilde{x})>0$, and (iii) $q_{1}(\tilde{x})<0$.

1. (i)

If $q_{1}(\tilde{x})=0$ then $q_{0}(\tilde{x})=\tilde{f}(\tilde{x})\leq\widetilde{\operatorname{Opt}}+\epsilon_{c}/2\leq\operatorname{Opt}+\epsilon_{c}\leq\operatorname{Opt}+\epsilon_{r}$. 2. (ii)

Now suppose $q_{1}(\tilde{x})>0$, i.e., we are in step 4 of Algorithm 4. We will need an upper bound on the value of $\alpha$ found in step 4.(d).

Let $t\coloneqq q(\tilde{\gamma}_{+},\tilde{x})$. Recall that $\lambda_{\min}(A(\tilde{\gamma}_{+}))\in[0,\delta/\kappa]$ (see Lemma 2). Then, as we have conditioned on the ApproxEig call in step 4.(a) succeeding, we have

[TABLE]

Next, we give a lower bound on $q(\tilde{\gamma}_{-},\tilde{x}+\alpha d)-(t+\alpha e)$ using the estimate $d^{\top}A(\tilde{\gamma}_{-})d\geq\xi$, and routine estimates on $\left\lVert A(\gamma)\right\rVert$ and $\left\lVert b(\gamma)\right\rVert$:

[TABLE]

where the last inequality follows from the bounds $\left\lvert\tilde{\gamma}_{+}-\tilde{\gamma}_{-}\right\rvert\leq\gamma_{+}\leq\zeta$ (Definition 1), $\left\lVert\tilde{x}\right\rVert\leq 6\kappa$ (Lemma 14), and Lemma 12.

We may combine our upper and lower bounds to deduce that for any $\alpha\in{\mathbb{R}}$,

[TABLE]

where the last relation follows from the definition of $\delta$ in (10), the definition of $\epsilon_{c}$ and the assumption on $\epsilon_{r}$, we have $\epsilon_{c}\leq\kappa^{2}\xi$, and the bound $\kappa\geq 1$. In particular, because the quadratic function on the left is negative at $\alpha=0$ and is lower bounded by a strongly convex quadratic function, there must exist both positive and negative choices of $\alpha$ for which the left hand side takes the value zero. This justifies step 4.(d) of the algorithm.

We now fix $\alpha$ to be the positive solution to $q(\tilde{\gamma}_{-},\tilde{x}+\alpha d)=q(\tilde{\gamma}_{+},\tilde{x}+\alpha d)$ so that

[TABLE]

We get an upper bound on $\alpha$ by the quadratic formula

[TABLE]

Then, by defining $\bar{x}\coloneqq\tilde{x}+\alpha d$, we have $q(\tilde{\gamma}_{-},\bar{x})=q(\tilde{\gamma}_{+},\bar{x})$. Note that the containment $\hat{\gamma}\in(\tilde{\gamma}_{-},\tilde{\gamma}_{+})$ from (12) implies $\tilde{\gamma}_{-}\neq\tilde{\gamma}_{+}$. Then we deduce $q_{1}(\bar{x})=0$. Moreover, our upper bound (15) gives

[TABLE]

Then recalling that $t\coloneqq q(\tilde{\gamma}_{+},\tilde{x})\leq\widetilde{\operatorname{Opt}}+\epsilon_{c}/2\leq\operatorname{Opt}+\epsilon_{c}$ and that we picked $e\leq 0$, we bound

[TABLE] 3. (iii)

The final case is symmetric to case (ii) and is omitted.

The running time of this algorithm follows from Lemma 11 and Theorem 4. ∎

Remark 12**.**

Let us now compare the running time of our algorithms to the running time of the algorithm presented by Jiang and Li [18]. This algorithm takes as input a pair $q_{0}$, $q_{1}$ satisfying an assumption similar to our Assumption 4 and a regularity parameter $\xi_{\text{JL}}$. See Remark 8 for a discussion of how the parameter $\xi_{\text{JL}}$ relates to our regularity parameters $(\xi_{\text{us}},\zeta)$. Then given $\epsilon>0$ and $p>0$, this algorithm returns an $\epsilon$-optimal feasible solution with probability at least $1-p$. The running time of this algorithm is

[TABLE]

where $\phi$ is a computable regularity parameter.

Recall that in Remark 8, we constructed simple examples where $\xi_{\text{JL}}\approx 1/2\kappa$ and $\zeta\approx 1$. One can check that the regularity parameter $\phi$ is a constant on these examples. In particular, the analysis presented in Jiang and Li [18] implies a running time of

[TABLE]

on these instances. We contrast this with the running times

[TABLE]

of our Algorithms 1 and 4 for finding an $\epsilon$-optimal value, and an $\epsilon$-optimal feasible solution respectively on these instances.

Remark 13**.**

Algorithms 1 and 4 were designed and analyzed with worst-case guarantees in mind. Consequently, we have not been particularly careful about bounding the constants in our analysis (for example the bounds $\kappa,\zeta,\xi^{-1}\leq 1$ are routinely used). As such, there may be variants of our algorithms that achieve the same worst-case guarantees with significantly faster numerical performance. Similarly, the algorithm presented by Jiang and Li [18] is analyzed with worst-case guarantees in mind. They also remark that the the numerical performance of their algorithm may improve “with suitable modifications” (see Jiang and Li [18, Remark 4.2]).

We leave such implementation questions and a thorough comparison of the numerical performance of the algorithms present in the literature for future work.

4.5 Further remarks

The algorithms given in the prior subsections can be sped up substantially if we know how to compute $\gamma_{\pm}$ and the corresponding zero eigenvectors exactly. As an example, we consider the special case where $A_{0}$ and $A_{1}$ are diagonal matrices.

Lemma 15**.**

There exists an algorithm which given $q_{0}$, $q_{1}$ satisfying Assumption 4 with $A_{0}$ and $A_{1}$ diagonal, returns $\gamma_{\pm}$, $(\xi^{*},\zeta^{*})$ and $\gamma^{*}$ such that $\lambda_{\min}(A(\gamma^{*}))=\xi^{*}$ in time $O(n)$.

Proof.

Let $a_{0}$, $a_{1}\in{\mathbb{R}}^{n}$ be the diagonal entries of $A_{0}$ and $A_{1}$ respectively. Note that

[TABLE]

Thus, $\gamma_{\pm}$ and $\zeta^{*}$ can clearly be computed in $O(n)$ time. Note that

[TABLE]

Hence, $\xi^{*}$ and $\gamma^{*}$ are, respectively, the optimal value and solution to a two-variable linear program with $n$ constraints. Applying the algorithm by Megiddo [23] for two-variable linear programming allows us to solve for $\xi^{*}$ and $\gamma^{*}$ in $O(n)$ time. ∎

Corollary 5**.**

There exists an algorithm which given $q_{0}$, $q_{1}$ satisfying Assumption 4 with $A_{0}$ and $A_{1}$ diagonal and error parameter $\epsilon>0$, outputs $\bar{x}\in{\mathbb{R}}^{n}$ such that

[TABLE]

This algorithm runs in time

[TABLE]

Proof.

When $A_{0}$ and $A_{1}$ are diagonal we have $N\leq 2n$. By Lemma 15, we can compute all of the quantities needed for the exact convex reformulation in $O(n)$ time. Algorithm 3 can then be applied to the exact convex reformulation to find $\tilde{x}\in{\mathbb{R}}^{n}$ with

[TABLE]

We can further carry out the modification procedure of Lemma 3 exactly in $O(n)$ time.

The running time of this algorithm follows from Corollary 4. ∎

Acknowledgments

This research is supported in part by NSF grant CMMI 1454548.

Appendix A Proofs of Theorems 2.3 and 2.3

In this appendix, we outline how to modify the proofs of Theorems 1 and 2 to prove Theorems 2.3 and 2.3.

\pdGeneralConvHull

Proof.

The “$\subseteq$” inclusions follow from a trivial modification of Lemma 2. It suffices to prove the “$\supseteq$” inclusions. The case where $A_{0}$ and $A_{1}$ are both nonconvex is covered by Theorem 1. We consider the four remaining cases:

• •

Suppose $A_{0}$ and $A_{1}$ are both convex. In this case, $\Gamma=[0,\infty)$ and it suffices to show that $\operatorname{conv}({\cal S})=\left\{(x,t):\,q_{0}(x)\leq t,\,q_{1}(x)\leq 0\right\}={\cal S}$. This holds as ${\cal S}$ is convex.

• •

Suppose $A_{0}$ is nonconvex and $A_{1}$ is convex. In this case, $\Gamma=[\gamma_{-},\infty)$ is unbounded above. Furthermore, $\gamma_{-}$ is positive and $A(\gamma_{-})$ has a zero eigenvalue. Suppose $(\hat{x},\hat{t})$ satisfies $q(\gamma_{-},\hat{x})\leq\hat{t}$ and $q_{1}(\hat{x})\leq 0$. If $q_{1}(\hat{x})=0$, then we also have $q_{0}(\hat{x})=q(\gamma_{-},\hat{x})\leq\hat{t}$, whence $(\hat{x},\hat{t})\in{\cal S}$. On the other hand, if $q_{1}(\hat{x})<0$, we may apply the argument in case (iii) in the proof of Lemma 3 verbatim (after replacing all occurrences of $\gamma_{+}$ by $\gamma^{*}$) to conclude that $(\hat{x},\hat{t})\in\operatorname{conv}({\cal S})$.

• •

Suppose $A_{0}$ is convex and $A_{1}$ is nonconvex. In this case, $\Gamma=[0,\gamma_{+}]$ is bounded above and $\gamma_{-}$ is defined to be $\gamma_{-}=0$. Furthermore, $A(\gamma_{+})$ has a zero eigenvalue. Suppose $(\hat{x},\hat{t})\in{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$. If $q_{1}(\hat{x})\leq 0$, then we also have $q_{0}(\hat{x})=q(\gamma_{-},\hat{x})\leq\hat{t}$, whence $(\hat{x},\hat{t})\in{\cal S}$. On the other hand, if $q_{1}(\hat{x})>0$, we may apply the argument in case (ii) in the proof of Lemma 3 verbatim to conclude that $(\hat{x},\hat{t})\in\operatorname{conv}({\cal S})$.∎

We will prove Theorem 2.3 using a limiting argument and reducing it to Theorem 2.3. The proof follows that of Lemma 5 almost verbatim.

\psdGeneralConvHull

Proof.

The “$\subseteq$” inclusions follow from a trivial modification of Lemma 4. It suffices to prove the “$\supseteq$” inclusions.

Denote the set on the right hand side by ${\cal R}$, i.e., ${\cal R}\coloneqq{\cal S}(\gamma_{-})\cap{\cal S}(\gamma_{+})$ when $\Gamma$ is bounded and ${\cal R}\coloneqq{\cal S}(\gamma_{-})\cap\left\{(x,t):\,q_{1}(x)\leq 0\right\}$ when $\Gamma$ is unbounded.

Let $(\hat{x},\hat{t})\in{\cal R}$. It suffices to show that $(\hat{x},\hat{t}+\epsilon)\in\operatorname{conv}({\cal S})$ for all $\epsilon>0$.

We will perturb $A_{0}$ slightly to create a new instance of the problem. Let $\delta>0$ to be picked later. Define $A_{0}^{\prime}=A_{0}+\delta I_{n}$ and let all remaining data be unchanged, i.e.,

[TABLE]

We will denote all quantities related to the perturbed system with an apostrophe.

We claim it suffices to show that there exists $\delta>0$ small enough such that $(\hat{x},\hat{t}+\epsilon)\in{\cal R}^{\prime}$. Indeed, suppose this is the case. Note that for any $x\in{\mathbb{R}}^{n}$, we have $q_{1}(x)=q_{1}^{\prime}(x)$ and $q_{0}(x)\leq q_{0}^{\prime}(x)$. Hence, $\operatorname{conv}({\cal S}^{\prime})\subseteq\operatorname{conv}({\cal S})$. Then, noting that $A^{\prime}(\gamma^{*})=A(\gamma^{*})+\delta I_{n}\succ 0$, we may apply Theorem 2.3 to the perturbed system to get $(\hat{x},\hat{t}+\epsilon)\in{\cal R}^{\prime}=\operatorname{conv}({\cal S}^{\prime})\subseteq\operatorname{conv}({\cal S})$ as desired.

First note that $A_{1}=A_{1}^{\prime}$ so that $\Gamma$ is bounded if and only if $\Gamma^{\prime}$ is bounded. We will then pick $\delta>0$ small enough such that

[TABLE]

where the last condition is only required when $\gamma_{+}$ and $\gamma_{+}^{\prime}$ both exist. This is possible as the expression on the left of each inequality is continuous in $\delta$ and is strictly satisfied when $\delta=0$.

The following computation shows that $q^{\prime}(\gamma_{-}^{\prime},\hat{x})\leq\hat{t}+\epsilon$.

[TABLE]

The first inequality follows by noting $q^{\prime}(\gamma,x)=q(\gamma,x)+\delta\left\lVert x\right\rVert^{2}$, the second inequality follows from our assumptions on $\delta$, and the third inequality follows from the assumption that $(\hat{x},\hat{t})\in{\cal S}(\gamma_{-})$. Thus $(\hat{x},\hat{t}+\epsilon)\in{\cal S}^{\prime}(\gamma_{-}^{\prime})$. When $\Gamma$ is bounded (or equivalently, when $\gamma_{+}^{\prime}$ and $\gamma_{+}$ exist), a similar calculation shows that $q^{\prime}(\gamma_{+}^{\prime},\hat{x})-(\hat{t}+\epsilon)\leq 0$ so that $(\hat{x},\hat{t}+\epsilon)\in{\cal S}^{\prime}(\gamma^{\prime}_{+})$. Finally, when $\Gamma$ is unbounded we have $q_{1}^{\prime}(\hat{x})=q_{1}(\hat{x})\leq 0$ so that $(\hat{x},\hat{t}+\epsilon)\in\left\{(x,t):\,q_{1}^{\prime}(x)\leq 0\right\}$. Thus, $(\hat{x},\hat{t}+\epsilon)$ is in ${\cal R}^{\prime}$, concluding the proof. ∎

Appendix B Estimation of the regularity parameters

In Section 4 we gave algorithms to solve the GTRS assuming that we had access to $(\xi,\zeta)$ and $\hat{\gamma}$ satisfying Assumption 5. In this appendix, we show how to compute these quantities.

Let $q_{0},q_{1}$ satisfy Assumption 4. Recall the definitions

[TABLE]

We will find $(\xi,\zeta)$ satisfying

[TABLE]

and a $\hat{\gamma}$ such that $\lambda_{\min}(A(\hat{\gamma}))\geq\xi$.

We will accomplish this in two stages. We begin by estimating $\xi^{*}$ using only an upper bound $\bar{\zeta}$ of $\zeta^{*}$. Then using our estimate $\xi$ we will compute $\zeta$.

B.1 Computing $\xi$ and $\hat{\gamma}$

We start with the following guarantee for the algorithm TestXi (Algorithm 5).

Lemma 16**.**

Given $q_{0},q_{1}$ satisfying Assumption 4, an arbitrary $0<\xi\leq 1$, an upper bound $\bar{\zeta}\geq\zeta^{*}$, and a failure probability $p_{\xi}>0$, TestXi (Algorithm 5) will output

[TABLE]

with probability $1-p_{\xi}$. This algorithm runs in time

[TABLE]

Proof.

We condition on the event that ApproxEig succeeds every time it is called. By the union bound, this happens with probability at least $1-p_{\xi}$.

As we have conditioned on ApproxEig succeeding, any $\hat{\gamma}$ which is output by TestXi will satisfy

[TABLE]

It is clear that TestXi will output “Fail” if $\xi>2\xi^{*}$ as there does not exist any $\hat{\gamma}$ such that $\lambda_{\min}(A(\hat{\gamma}))\geq\xi^{*}$. It remains to show that, given $\xi\leq\xi^{*}$, TestXi will output some $\hat{\gamma}$.

For the sake of contradiction, assume that the algorithm fails to output in each of the $T$ rounds. Let $P\coloneqq\left\{\gamma:\,\lambda_{\min}(A(\gamma))\geq 3\xi^{*}/4\right\}$. Recall that $\lambda_{\min}(A(\gamma))$ is $1$-Lipschitz in $\gamma$. As there exists some $\gamma$ such that $\lambda_{\min}(A(\gamma))\geq\xi^{*}$ (see Definition 1), we conclude that $P$ is an interval of length at least $\xi^{*}/2$.

Note that $P\subseteq[s_{0},t_{0}]$. We will inductively show that $P\subseteq[s_{k},t_{k}]$ for each $k\in\left\{1,\dots,T\right\}$. Let $k\in\left\{0,\dots,T-1\right\}$ and let $s_{k},\bar{\gamma},t_{k}$ be defined as in the algorithm and let $x$ be the unit vector found in step 3.(d). We claim that $x^{\top}A_{1}x\neq 0$. Indeed suppose $x^{\top}A_{1}x=0$, then $x^{\top}A(\gamma)x=x^{\top}A(\bar{\gamma})x\leq 3\xi/4$ for all $\gamma$. This contradicts the assumption that there exists some $\gamma$ such that $\lambda_{\min}(A(\gamma))\geq\xi$. Now suppose $\gamma\in P$, then

[TABLE]

where the first inequality follows from $\gamma\in P$, and the last one from the fact that the algorithm did not output in iteration $k$ (and thus the if statement in step 3.(d) did not hold). Thus, if $x^{\top}A_{1}x>0$, then we have the implication $\gamma\in P\implies\gamma\geq\bar{\gamma}$. Similarly, if $x^{\top}A_{1}x<0$, then we have the implication $\gamma\in P\implies\gamma\leq\bar{\gamma}$. Then by induction, we have $P\subseteq[s_{k+1},t_{k+1}]$.

We conclude that $P$, an interval of length at least $\xi^{*}/2$, is contained in $[s_{T},t_{T}]$ an interval of length

[TABLE]

Noting that $\xi\leq\xi^{*}$ gives us the desired contradiction.

The running time of this algorithm follows from Lemma 11. ∎

Given a lower bound $\xi\leq\xi^{*}$, Lemma 16 guarantees that TestXi will find a $\hat{\gamma}$ satisfying $\lambda_{\min}(A(\hat{\gamma}))\geq\xi/2$ with high probability. In order to make use of this lemma without a lower bound on $\xi^{*}$, we will simply repeatedly call TestXi with decreasing guesses for $\xi$. Consider Algorithm 6.

Theorem 6**.**

Given $q_{0},q_{1}$ satisfying Assumption 4, an upper bound $\bar{\zeta}\geq\zeta^{*}$, and a failure probability $p>0$, ApproxXi (Algorithm 6) will output $\xi$ and $\hat{\gamma}$ such that

[TABLE]

and run in time

[TABLE]

with probability $1-p$.

Proof.

We condition on the event that TestXi succeeds every time it is called. By the union bound, this happens with probability at least $1-p$.

Let $k^{*}\in\left\{1,2,\dots\right\}$ be such that $\xi^{*}/2\leq 2^{-k^{*}}<\xi^{*}$. Then, as we have conditioned on TestXi succeeding, Lemma 16 guarantees that TestXi$(q_{0},q_{1},2^{-k},\bar{\zeta},2^{-(k+1)}p)$ outputs

[TABLE]

Thus, TestXi will output “Fail” for every $k<k^{*}$ and will output $\hat{\gamma}$ either on round $k^{*}$ or $k^{*}+1$. We can then bound

[TABLE]

We bound the run time of the algorithm as follows.

[TABLE]

B.2 Computing $\zeta$

Recall the guarantee of the algorithm ApproxGammaPlus. \ApproxGammaPlusLemma*

We will repeatedly call ApproxGammaPlus with different choices of $\delta$. Consider the algorithm ApproxZeta.

Theorem 7**.**

Given $q_{0},q_{1}$ satisfying Assumption 4, $(\xi,\bar{\zeta})$ and $\hat{\gamma}$ satisfying Assumption 5, and failure probability $p>0$, ApproxZeta (Algorithm 7) will output $\zeta$ such that

[TABLE]

and run in time

[TABLE]

with probability $1-p$.

Proof.

We condition on the event that ApproxGammaPlus succeeds every time it is called. By the union bound, this happens with probability at least $1-p$.

We first check that the assumptions of Lemma 2 hold. For $k=1$, we have $2^{-(k-1)}\bar{\zeta}=\bar{\zeta}\geq\zeta^{*}$. Then by induction, and conditioning on ApproxGammaPlus succeeding, Lemma 2 guarantees

[TABLE]

If ApproxZeta fails to terminate in round $k$, then 1.(b) ensures $\hat{\zeta}_{k}\leq 2^{-(k+1)}\bar{\zeta}$. This in turn implies that $\zeta^{*}\leq 2^{-((k+1)-1)}\bar{\zeta}$ and, by induction, the assumptions of Lemma 2 hold in every round that ApproxGammaPlus is called.

Let $k$ be the round in which the algorithm terminates. If $k=1$, then the guarantee of Lemma 2 implies $\zeta^{*}\geq\hat{\zeta}_{1}$, whence

[TABLE]

Thus, we may assume $k\geq 2$. The condition of step 1.(b) then guarantees the two inequalities

[TABLE]

Then, we have

[TABLE]

where the first and fifth relations follow from Lemma 2 and the second and fourth relations follow from (16) above.

It remains to bound the run time of ApproxZeta. Let $k^{*}\in\left\{1,2,\dots\right\}$ be such that $\zeta^{*}\leq 2^{-(k^{*}-1)}\bar{\zeta}<2\zeta^{*}$. We show that ApproxZeta terminates within $k^{*}$ rounds. Suppose ApproxZeta reaches the $k^{*}$th round. Then, we have

[TABLE]

where we used Lemma 2 in the first relation, and the definition of $k^{*}$ in the second relation. Therefore, ApproxZeta terminates in round $k^{*}$ at the latest and we can bound the run time of this algorithm as

[TABLE]