Exact Optimization: Part I

TL;DR

This paper introduces a novel geometric approach to nonlinear programming, providing explicit algebraic solutions to quadratic programming problems without requiring initial feasible points.

Contribution

It offers a new perspective based on convex quadratic equations, enabling derivative-free, explicit solutions to various quadratic programming problems.

Findings

Closed-form solutions for QPs with various constraints

Unified parameterization of solutions to CQE

Enhanced understanding of CQF and CQE relations

Abstract

Nonlinear programming is explicitly analyzed via a novel perspective/method and from a bottom-up manner. The philosophy is based on the recent findings on convex quadratic equation (CQE), which help clarify a geometric interpretation that relates CQE to convex quadratic function (CQF). More specifically, regarding the solvability of CQE, its necessary and sufficient condition as well as a unified parameterization of all the solutions has recently been analytically formulated. Moving forward, the understanding of CQE is utilized to describe the geometric structure of CQF, and the CQE-CQF relation. All these results are shown closely related to a basis in the optimization literature, namely quadratic programming (QP). For the first time from this viewpoint, the QPs subject to equality, inequality, equality-and-inequality, and extended constraints can be algebraically solved in derivative-free closed formulae, respectively. All the results are derived without knowing a feasible point, a priori and any time during the process.

Equations

$$F_{\mathbf{x}}(\mathbf{x})={\mathbf{x}}^{T}P\mathbf{x}/2+{\mathbf{q}}^T\mathbf{x}+s,$$

$${\mathbf{z}}^{T}M\mathbf{z}+{\mathbf{k}}^T\mathbf{z}+c=0,$$

$${\mathbf{k}}^T{M}^{-1}\mathbf{k}\ge 4c.$$

$$\mathbf{z}=-M^{-1}\mathbf{k}/2+\sqrt{{\mathbf{k}}^T{M}^{-1}\mathbf{k}/4-c}\cdot M^{-1/2}\cdot \mathbf{v},$$

$$\mathbf{k}\in \mathcal{R}(M)\text{mbox}and{\mathbf{k}}^T{M}^{†}\mathbf{k}\ge 4c,$$

$$\mathbf{k}\notin \mathcal{R}(M).$$

$$\mathbf{z}=-M^{†}\mathbf{k}/2+\sqrt{{\mathbf{k}}^T{M}^{†}\mathbf{k}/4-c}\cdot M^{†/2}\mathbf{\rho }+\mathbf{\epsilon },$$

$$\mathbf{z}=-(F_{\mathbf{w}}/||{\mathbf{k}}_{M^{\perp }}|{|}^2)\cdot {\mathbf{k}}_{M^{\perp }}+\mathbf{\phi }+\mathbf{\tau },$$

$$F_{\mathbf{w}}(\mathbf{w})={\mathbf{w}}^{T}M\mathbf{w}+{\mathbf{k}}_M^T\mathbf{w}+c,$$

$$\text{mbox}minimize{F}_{\mathbf{x}}\text{mbox}subjecttoA\mathbf{x}=\mathbf{b},$$

$$\mathbf{x}=A^{†}\mathbf{b}+V_2\mathbf{y},$$

$${\bar{l}}^{\ast }=[{\mathbf{b}}^T(A^{†}{)}^{T}P/2+{\mathbf{q}}^T][I_n-V_2(V_2^{T}P{V}_2{)}^{†}{V}_2^{T}P]A^{†}\mathbf{b}+s-{\mathbf{q}}^T{V}_2(V_2^{T}P{V}_2{)}^{†}{V}_2^T\mathbf{q}/2,$$

$${\mathbf{x}}^{\bar{\ast }}={\mathbf{x}}_p^{\bar{\ast }}+V_2{\mathbf{\epsilon }}^{\bar{\ast }},$$

$${\bar{l}}_{ln}^{\ast }$$

$${\mathbf{x}}_{ln}^{\bar{\ast }}$$

$$\text{mbox}minimize{F}_{\mathbf{x}}$$

$$\text{mbox}subjecttoA\mathbf{x}=\mathbf{b},$$

$${\mathbf{c}}_i^T\mathbf{x}\le d_i,i\in \mathcal{I}=\{1,2,\cdots ,\kappa \},$$

$$\text{mbox}minimize{F}_{\mathbf{x}}\text{mbox}subjecttoboth\mathbf{x},\mathbf{q}\in \mathcal{R}(P),$$

$$\mathbf{x}=-P^{†}\mathbf{q}+\sqrt{{\mathbf{q}}^T{P}^{†}\mathbf{q}-2s+2\check{l}}\cdot P^{†/2}\cdot \check{\mathbf{\rho }},$$

$$\mathbf{w}=-M^{†}{\mathbf{k}}_M/2+\sqrt{{\mathbf{k}}_M^T{M}^{†}{\mathbf{k}}_M/4-c+{\breve{F}}_{\mathbf{w}}}\cdot M^{†/2}\cdot \hat{\mathbf{\rho }},$$

$$\displaystyle\mbox{det}(\left[\begin{array}[]{c:c}\begin{array}[]{c}{\bf c}_{1}^{T}{}\\ {\bf c}_{2}^{T}{}\\ {\bf c}_{3}^{T}{}\end{array}&\begin{array}[]{c}~{}d_{1}\\ ~{}d_{2}\\ ~{}d_{3}\end{array}\end{array}\right])\neq 0.$$

$$F_{\mathbf{x}}(\mathbf{x})={\Vert P^{1/2}\mathbf{x}+P^{-1/2}\mathbf{q}\Vert }^2/2+s-{\mathbf{q}}^T{P}^{-1}\mathbf{q}/2,$$

$$F_{\mathbf{x}}(\mathbf{x})={\mathbf{x}}_1^T{\breve{\Sigma }}_1{\mathbf{x}}_1+{\mathbf{q}}_1^T{\mathbf{x}}_1+s+{\mathbf{q}}_2^T{\mathbf{x}}_2,$$

$$F_{{\mathbf{x}}_1}({\mathbf{x}}_1)={\mathbf{x}}_1^T{\breve{\Sigma }}_1{\mathbf{x}}_1+{\mathbf{q}}_1^T{\mathbf{x}}_1+s,$$

$$\displaystyle A=W\cdot\left[\begin{array}[]{cc}\Gamma_{1}{}{}&0\end{array}\right]\left[\begin{array}[]{c}V_{1}^{T}\\ V_{2}^{T}\end{array}\right],$$

$$A^{†}$$

$$=$$

$$F_{\mathbf{y}}(\mathbf{y})={\mathbf{y}}^T{V}_2^{T}P{V}_2\mathbf{y}/2+[{\mathbf{b}}^T(A^{†}{)}^{T}P+{\mathbf{q}}^T]V_2\mathbf{y}+s+[{\mathbf{q}}^T+{\mathbf{b}}^T(A^{†}{)}^{T}P/2]A^{†}\mathbf{b}.$$

$$P={\check{U}}_1{\check{\Sigma }}_1{\check{U}}_1^T\text{mbox}(full/thinversion),$$

$$\Vert {\check{\Sigma }}_1^{1/2}{\check{U}}_1^T{V}_2\tilde{\mathbf{y}}{\Vert }^2>0$$

$$\iff {\check{U}}_1^T\cdot V_2\tilde{\mathbf{y}}\ne 0$$

$$\iff \mathcal{N}(P)\cap \mathcal{N}(A)=\{0\},$$

Full text

I Introduction

Recently, [14] provided more understanding of the mathematical fundamentals: convex quadratic equation and function (CQE and CQF). In particular, [14] has analytically and completely formulated an equivalent solvability condition of CQE and a parameterization of all its solutions. These results encourage new investigations into a spectrum of applications, such as the one to nonlinear optimal control ([14, Sec. 4]). The findings not only facilitate preliminary optimality recovery using the state-dependent Riccati equation (SDRE) scheme and its differential variant (see [3] for a survey), but also enhance the computational performance of the other major application in this article. Specifically, we analyze the nonlinear programming [16]/convex optimization [2] from a new perspective based on [14], which particularly boasts impacts in management science, operations research, and control engineering, for example, the model predictive control (MPC) [4]. It is worth mentioning that [10] and [22] reveal more connection between the two applications, that is, nonlinear MPC using “state-dependent coefficient” (a design flexibility) in the SDRE scheme. Such a unified framework is envisioned/pioneered by [1], and also pursued by this article.

With regard to the convex optimization, a basis in the literature of nonlinear programming algorithms is the quadratic programming (QP) [16], which includes the linear programming (LP) as a special case and is closely linked with the CQE and CQF [21, 24]. Regarding the former reference, [21] utilize a solution subset of CQE, which is reformulated from the explicitly considered CQF and in terms of the polyhedral characterization [17]. Considering the latter, [24] make use of an important equality that is related with a CQF, as well as the existence and uniqueness of the corresponding optimum (point), with respect to a QP-like optimization problem. This supports the construction of a feasible sequence converging to the optimum. Generally speaking, more iterative/numerical QP solvers are available in the mainstream literature [20], which are mostly analyzed from the differential perspective and can be classified by three levels [2]: i) equality-constrained QP; ii) linear equality-constrained optimization problem with twice-differentiable objective, as solved using Newton’s method by reducing it to a sequence of equality-constrained QPs; and, after further imposing inequality constraints, iii) interior-point methods, which reduce the problem to a sequence of ii). It is worth noting that the equality-constrained QP solver is the most fundamental to build up various algorithms, subject to the constraints at the three levels. In addition, one area of great creativity among existing algorithms, namely the conjugate direction methods, illustrates that detailed analysis of QP leads to significant practical advances [16], such as solution accuracy, computation time, and more reported in [9] (see Sec. III-E later for a comparison discussion). Moreover, the constrained QP can also be regarded as an optimization problem with the objective of constrained convex quadratic function (CCQF). These observations quite endorse the importance of QP and CQF, while motivate this article to provide further theoretical support (extended from [14]). One highlight is the first closed-form QP solver subject to a variety of constraints. The analytical philosophy is inspired by [15] (vector space methods) as well as the computational interest among a variety of research fields [27, 5, 8, 16, 23]; but based on a novel understanding and perspective of the associated CQE, and interpretations of its relations to CQF and QP, respectively.

Following the previous findings of CQE and CQF [14], this article deals with another application to nonlinear programming and – in response to, for example, the open question in [5, Sec. VI] – the main result is exactly, autonomously, and algebraically solving the QPs that

• •

are subject to equality, inequality, equality-and-inequality, and extended constraints, respectively;

• •

refrain from any derivative (only functional information is used);

• •

consist of finite steps, the maximum of which is explicitly pre-determined;

• •

and do not require any knowledge of a feasible point (any time before and during the process).

The derivation leverages the vector space methods such as the singular value decomposition (SVD), and investigates from a perspective based on [14] and geometric interpretations that clarify the structures of CQE, CQF, and CQE-CQF relation, respectively.

The rest of this article is arranged as follows. The notations and the formulation of main problem are provided in Sec. II. The method proposed in [14] is applied to this focused application “nonlinear programming/convex optimization” in Sec. III. At first, the analysis of extreme/minimal values of CQF is performed in Sec. III-A, which is followed by a complete characterization of the explicit solution to the equality-constrained QP in Sec. III-B. As a step forward, the two preliminary subsections are shown indispensable for the proposed closed-form QP solver in Sec. III-C, and an extension subject to the constraints in Sec. III-D. Therefore, discussions on differences and advantages with respect to existing solvers in the literature can be justified in Sec. III-E. Finally, three representative examples are demonstrated to offer computational evidences and additional insights (as verified using MATLAB®) in Sec. IV, while concluding remarks are given in Sec. V.

II Notation and Problem Formulation

We use the notational conventions, unless noted otherwise: the symbols $||\cdot||$, ${\mathcal{N}}(\cdot),~{}{\mathcal{R}}(\cdot),~{}\lambda(\cdot),~{}(\cdot)^{\dagger},~{}(\cdot)^{\dagger/2}$, and $(\cdot)^{T}$ denote the Euclidean norm, null space, range space, spectrum/eigenvalues, Moore-Penrose generalized inverse (shortened as “pseudoinverse” afterward), square root of the pseudoinverse [26], and transpose of a vector or matrix, respectively. Additionally, we denote $(\cdot)^{\perp}$ as the orthogonal complement of a vector space, $\oplus$ the direct sum of vector spaces, ${\bf e}_{1}$ the first standard basis vector in n, and $\{\sigma_{i}\}_{i=1}^{\it n}$ the set of singular values of an ${\it n}\times{\it n}$ matrix with rank ${\it r}$ ($\sigma_{1}\geq\sigma_{2}\geq\cdots\geq\sigma_{\it n}\geq 0$ and $\sigma_{\it r}>0$). Moreover, denote $P\succ 0$ (resp., $P\succeq 0$), if $P=P^{T}\in{}^{{\it n}\times{\it n}}$ is positive definite (resp., semidefinite) [16].

Consider the following CQF $F_{\bf x}:{}^{\it n}\rightarrow\real$ [16],

[TABLE]

where both ${\bf x},~{}{\bf q}\in{}^{\it n}$, $P=P^{T}\in{}^{{\it n}\times{\it n}}$, $P\succeq 0$, and $s\in\real$. In particular, the function (1) is strictly convex, if its Hessian matrix $P\succ 0$. Note that the early findings in [14] start with the formulation of CQE, as in Eq. (2) later, to emphasize its dominance. Then, in this continuing work, the concept of CQF dominates, and thus we also give its formulation in the very beginning (1) and interpret its close relation to CQE later in detail. More specifically, in this article, the main focus shifts to explicitly analyzing and solving the following problem.

Problem 1

From a new viewpoint based on the solution to [14, Problem 2.1] as well as the CQE-CQF relation, solve the QPs in an analytical, autonomous, and derivative-free manner and subject to equality, inequality, equality-and-inequality, and extended constraints, respectively.

III Application to Convex Optimization (QP)

The main concepts behind all the derivations in this section are the geometrical interpretations of i) the value/difference “${\bf q}^{T}P^{\dagger}{\bf q}/2-s$” according to the solvability conditions in Eqs. (3) and (5) later and ii) the solution parameterization in Eqs. (4) and (7). Notably, in i), $P^{\dagger}=P^{-1}$ when $P$ is nonsingular. Moreover, the novelty can be revealed by that i) facilitates the derivations that exploit the (additional) analysis perspective from the image of the CQF (1). As a step forward, the hierarchical layers in the parameterization, with respect to ii), exactly categorize the location(s) of the unconstrained optimum/optima. In the following, Sec. III-A considers the unconstrained optimization, which are preliminaries supporting the derivations of constrained ones in the remaining subsections. Following the QP formulation/definition in literature [2, 16], Sec. III-D enlarges the included types of constraints, without introducing any excessive variable but, actually, reducing to a lower-dimensional unconstrained equivalent problem. Finally, Sec. III-E discusses advantages or differences over existing solvers [20]. Remarkably, we recall the following lemma because it is largely used in this article, where the notation is consistent with [14, 13] to avoid misleading/confusion.

Lemma 1

*(Solvability and Solutions of CQE [14, Theorem 3.1])

Consider the CQE below [16]:*

[TABLE]

where both ${\bf z},~{}{\bf k}\in\mathbb{R}^{\it n}$, $M=M^{T}\in\mathbb{R}^{{\it n}\times{\it n}}$, $M\succeq 0$, and $c\in\mathbb{R}$.

1. A)

If rank$(M)=n$, then CQE (2) is solvable, if and only if (iff)

[TABLE]

Accordingly, the set of solutions are, and can be parameterized by,

[TABLE]

where ${\bf v}\in\mathbb{R}^{\it n}$ and $||{\bf v}||=1$. 2. B)

Otherwise (rank$(M)<n$), it is solvable, iff (5) or (6), where

[TABLE]

Accordingly, the sets of solutions are, and can be parameterized by, respectively,

1. a)

for Condition (5),

[TABLE]

where both ${\boldsymbol{\rho}},~{}{\boldsymbol{\varepsilon}}\in\mathbb{R}^{\it n}$, ${\boldsymbol{\rho}}\in{\mathcal{R}}(M)$, $||{\boldsymbol{\rho}}||=1$, and ${\boldsymbol{\varepsilon}}\in{\mathcal{N}}(M)$; 2. b)

for Condition (6), decompose ${\bf k}={\bf k}_{M}+{\bf k}_{M^{\perp}}$, where ${\bf k}_{M}\in{\mathcal{R}}(M)$, ${\bf k}_{M^{\perp}}\in{\mathcal{R}}(M)^{\perp}$, and both ${\bf k}_{M},~{}{\bf k}_{M^{\perp}}\in\mathbb{R}^{\it n}$. Then,

[TABLE]

where the CQF $F_{\bf w}:{\mathcal{R}}(M)\subset\mathbb{R}^{\it n}\rightarrow\mathbb{R}$,

[TABLE]

all ${\bf w},~{}{\boldsymbol{\varphi}},~{}{\boldsymbol{\tau}}\in\mathbb{R}^{\it n}$, both ${\bf w},~{}{\boldsymbol{\tau}}\in{\mathcal{R}}(M)$, and ${\boldsymbol{\varphi}}\in{\mathcal{N}}(M)\cap{\mathcal{N}}({\bf k}^{T})$.

III-A Unconstrained QP

Theorem 1

*(Solutions to Unconstrained QP)

Consider the optimization problem, minimize $F_{\bf x}$ (1) with respect to ${\bf x}$.*

1. i)

The preimage of any level set of $F_{\bf x}$ (1) can be parameterized by Eqs. (4), (7)-(9), respectively, where $M=P/2$, ${\bf k}={\bf q}$, $c=s-l$, and $l\in\real$ is any level set value of $F_{\bf x}$. 2. ii)

The optimal value is finite, iff $~{}{\bf q}\in{\mathcal{R}}(P)$. 3. iii)

The finite optimal value is $l^{*}=s-{\bf q}^{T}P^{\dagger}{\bf q}/2$. And the corresponding unique optimum is, or optima are all parameterized by, ${\bf x}^{*}={\bf x}_{p}^{*}+\breve{\boldsymbol{\varepsilon}}$, where ${\bf x}_{p}^{*}\coloneqq-P^{\dagger}{\bf q}$, $\breve{\boldsymbol{\varepsilon}}\in{}^{\it n}$, and $\breve{\boldsymbol{\varepsilon}}\in{\mathcal{N}}(P)$.

Proof:

See Appendix A for a complete and unified proof, which is new with regard to ii) and iii), while first for i). Along with the proof, a geometric interpretation for the novel concept of “critical shift” (resp., “hierarchical layers”) is given in Fig. 1 (resp., Remark 12 and Fig. 2). ∎

Remark 1

Based on Lemma 1, the results of ii) and iii) in Theorem 1 are consistent with the literature, for example, [2] as analyzed from the differential perspective, which thus and also endorses Lemma 1. All the results in Theorem 1, particularly i), will be utilized for the constrained and extended QP problems, later in Secs. III-B to III-D. On the other hand, more potentials are expected toward the applications to least-squares approximation and regression analysis. Finally, according to the respective items in Theorem 1, it is worth remarking that:

1. i)

A geometric interpretation is illustrated in Fig. 1. 2. ii)

From its proof, typically Eq. (27), $F_{\bf x}$ is always unbounded above; bounded below, iff ${\bf q}\in{\mathcal{R}}(P)$. 3. iii)

Regarding the special but popular case of nonsingular $P$ **[16]**, then $P^{\dagger}=P^{-1}$, ${\mathcal{N}}(P)=\{{\bf 0}\}$, the unique optimum ${\bf x}^{*}=-P^{-1}{\bf q}$, and $l^{*}=s-{\bf q}^{T}P^{-1}{\bf q}/2$.

Remark 2

After introducing the concept of “critical shift $l^{*}$” in this section, as illustrated in Fig. 1, it suffices to complement the examination previously in [14, Remark 3.2]. By ii) and iii) of Theorem 1, the optimal/minimal value ($l^{*}$) is finite and equals zero (no shift) for $M\succeq 0$, either singular or nonsingular. In other words, ${\bf z}^{T}M{\bf z}>0$, for all ${\bf z}\not\in{\mathcal{N}}(M)$. To sum up, combining [14, Remark 3.2] and this Remark 2 endorse both Theorem 1 and Lemma 1 above, according to the basic mathematical properties of positive definiteness and semidefiniteness.

III-B Equality-Constrained QP

Theorem 2

*(Solutions to Equality-Constrained QP)

Consider the optimization problem,*

[TABLE]

where $F_{\bf x}$ is in Eq. (1), $A\in{}^{{\it m}\times{\it n}}$, rank$(A)={\it m}\in\{1,2,\cdots,{\it n}-1\}$, and ${\bf b}\in{}^{\it m}$. From the SVD of $A$, let a $V_{2}\in{}^{{\it n}\times({\it n}-{\it m})}$ be given, which is associated with ${\mathcal{R}}(V_{2})={\mathcal{N}}(A)$. Denote $\bar{l}\in\real$ as any level set value of $F_{\bf x}$.

1. A)

If $V_{2}^{T}PV_{2}\neq 0\Leftrightarrow{\mathcal{R}}(P)\cap{\mathcal{N}}(A)\neq\{{\bf 0}\}\Leftrightarrow{\mathcal{N}}(A)\not\subseteq{\mathcal{N}}(P)$,

1. a)

Problem (10) is equivalent to an unconstrained QP, 2. b)

the preimage of any level set/value of “$F_{\bf x}$* subject to $A{\bf x}={\bf b}$” can be respectively parameterized by*

[TABLE]

where ${\bf y}\in{}^{{\it n}-{\it m}}$ is determined by Eqs. (4), (7)-(9), with ${\bf z}={\bf y}$, $M=V_{2}^{T}PV_{2}/2$, ${\bf k}=V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})$, and $~{}c=s+[{\bf q}^{T}+{\bf b}^{T}(A^{\dagger})^{T}P/2]A^{\dagger}{\bf b}-\bar{l}$, 3. c)

among all $\bar{l}$ in A)b), the optimal one/minimum is finite, iff $~{}V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})\in{\mathcal{R}}(V_{2}^{T}PV_{2})$, 4. d)

following A)c), this finite optimal value is $\bar{l}^{*}\in\real$,

[TABLE]

and the corresponding unique optimum is, or optima are all, parameterized by,

[TABLE]

where ${\bf x}_{p}^{\bar{*}}\coloneqq A^{\dagger}{\bf b}-V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})\in{}^{\it n}$ denotes the particular solution of ${\bf x}^{\bar{*}}\in{}^{\it n}$, ${\boldsymbol{\varepsilon}}^{\bar{*}}\in{}^{{\it n}-{\it m}}$, and ${\boldsymbol{\varepsilon}}^{\bar{*}}\in{\mathcal{N}}(V_{2}^{T}PV_{2})$, 5. e)

to further categorize A)d), the optimum is unique, iff ${\mathcal{N}}(A)\cap{\mathcal{N}}(P)=\{{\bf 0}\}$; 2. B)

else, if $V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})\neq{\bf 0}$,

1. a)

Problem (10) is equivalent to an unconstrained LP, and thus unbounded, 2. b)

the preimage of any level set/value of “$F_{\bf x}$* subject to $A{\bf x}={\bf b}$” can be respectively parameterized by Eq. (11), where ${\bf y}$ is determined by [14, Lemma 3.1], with ${\bf z}={\bf y}$, ${\boldsymbol{\zeta}}=V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})$, and $\nu=s+[{\bf q}^{T}+{\bf b}^{T}(A^{\dagger})^{T}P/2]A^{\dagger}{\bf b}-\bar{l}$;* 3. C)

else,

1. a)

Problem (10) is equivalent to a constant function, 2. b)

its value is “$s+[{\bf q}^{T}+{\bf b}^{T}(A^{\dagger})^{T}P/2]A^{\dagger}{\bf b}$”, 3. c)

the (only) preimage is parameterized by Eq. (11), for all ${\bf y}$.

Proof:

See Appendix B for the first, complete, and unified proof. ∎

Remark 3

Both Theorems 1 and 2 present the results within a unified framework. In particular, the following summarize further analyses of the more significant, quadratic case A) in Theorem 2.

1. i)

Using the common terminology, such as **[2, Sec. 4.1.1]**, the considered problem is feasible since ${\bf b}\in{\mathcal{R}}(A)$. The preimage of any level set of “$F_{\bf x}$* (1) subject to $A{\bf x}={\bf b}$” is the feasible set. The definitions of the optimum/optima and corresponding optimal value follow the general consensus [2] throughout Sec. III, and thus omitted for brevity.* 2. ii)

Regarding the consistent constraint $A{\bf x}={\bf b}$ (in other words, ${\bf b}\in{\mathcal{R}}(A)$ where $A$ is of full rank) in this theorem, we omit the only remaining case (that is, rank$(A)={\it n}$) for a more concise presentation. This corresponds to ${\mathcal{N}}(A)=\{{\bf 0}\}$ and that there is only one feasible point ${\bf x}^{\bar{*}}=A^{\dagger}{\bf b}$. However, by an extended definition of $V_{2}\coloneqq 0_{\it n}$, the results of A)c)-A)e) in this theorem still apply to this sub-QP problem. 3. iii)

In the proof, the considered QP Problem (10) is equivalently reformulated as an unconstrained QP: minimize $F_{\bf y}$ (38) with respect to ${\bf y}$. This is viable, iff $V_{2}^{T}PV_{2}\neq 0$. With regard to the unconstrained QP, $F_{\bf y}$ (resp., the original QP Problem (10)) is always unbounded above; bounded below, iff $V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})\in{\mathcal{R}}(V_{2}^{T}PV_{2})$. Note that a geometric interpretation regarding the CQF $F_{\bf y}$ (resp., QP (10) for the optimal value) can be inferred from Fig. 1.

Remark 4

An example to endorse Theorem 2 comes from the minimum or least-norm problem [2, 6]. More specifically, to find the optimum ${\bf x}^{\bar{*}}_{ln}$ that (satisfies the consistent underdetermined constraint $A{\bf x}={\bf b}$ and) is of least $l_{2}$-norm, which (norm) is most commonly adopted. This is in the applicable form using this theorem, with $P=2I_{\it n}\succ 0$, ${\bf q}={\bf 0}$, and $s=0$. Accordingly, this is a QP given ${\mathcal{R}}(P)\cap{\mathcal{N}}(A)={\mathcal{N}}(A)$, which (null space) is at least one-dimensional and thus nonzero. Moreover, the optimal value (denoted $\bar{l}^{*}_{ln}$) is finite since $V_{2}^{T}A^{\dagger}{\bf b}=V_{2}^{T}V_{1}\Gamma_{1}^{-1}W^{T}{\bf b}={\bf 0}\in{\mathcal{R}}(I_{{\it m}})={}^{\it m}$, where both $V_{1}\in{}^{{\it n}\times{\it m}}$ being orthogonal to $V_{2}$ and $A^{\dagger}=V_{1}\Gamma_{1}^{-1}W^{T}\in{}^{{\it n}\times{\it m}}$ are given in Eq. (37) in the proof. Specifically,

[TABLE]

Correspondingly and similarly, the unique optimum is

[TABLE]

where ${\boldsymbol{\varepsilon}}^{\bar{*}}\in{\mathcal{N}}(V_{2}^{T}V_{2})=({}^{{\it n}-{\it m}})^{\perp}=\{{\bf 0}\}$. As expected, these results agree with the literature [2].

Remark 5

This remark focuses upon the special case of ${\it m}=1$, and more efficiently constructs an example of $V_{2}$ in Theorem 2. Without loss of generality, let $A={\bf a}^{T}\in{}^{1\times{\it n}}$ be normalized. Given the results in [14, Theorem 4.2] and further analyses in [14, Remark 4.4], $V_{2}$ is readily available by computing the last $({\it n}-1)$ columns of $H_{\boldsymbol{\iota}}^{\bf a}\coloneqq I_{\it n}-2{\boldsymbol{\iota}}_{\bf a}{\boldsymbol{\iota}}_{\bf a}^{T}$, where ${\boldsymbol{\iota}}_{\bf a}\coloneqq({\bf a}-{\bf e}_{1})/(\|{\bf a}-{\bf e}_{1}\|)$ and $Y=I_{{\it n}-1}$. This remark also indicates a research direction, which aims at extending to general ${\it m}$ in the equality-constrained QP (10) based on the generalization of [14, Theorem 4.2], and thus involves the orthogonalization on the rows of $A$ in the first place.

III-C QP

Consider the optimization problem [16],

[TABLE]

where $F_{\bf x}$ is in Eq. (1), $A\in{}^{{\it m}\times{\it n}}$, rank$(A)={\it m}\in\{1,2,\cdots,{\it n}-1\}$, ${\bf b}\in{}^{\it m}$, ${\bf c}_{i}\in{}^{\it n}$, $d_{i}\in\real$, and there exists a point that satisfies all the constraints.

To this general QP Problem (15), an explicit solver (namely, without any iterations, approximations, or assumptions as in the classical/numerical optimization [20, 16]) is available in Algorithm 1, whose proof is in Appendix C (together with geometric demonstrations via Figs. 3 and 4).

Algorithm 1

*Closed-Form QP Solver

 *

0:*  $P,~{}{\bf q},~{}s,~{}A,~{}{\bf b},~{}\{{\bf c}_{i},d_{i}\}_{i\in{\mathcal{I}}}$ as defined in (15)Initial Conditions: $\bar{\mathcal{L}}=\bar{\mathcal{X}}=\bar{\mathcal{I}}=\tilde{\mathcal{I}}^{*}=\emptyset$ *

0:*  The optimal value $\tilde{l}^{*}$ and the (resp., a subset of) corresponding optimum (resp., optima) ${\bf x}^{\tilde{*}}$ *

1:*   compute the SVD of $A$ till ${\mathcal{R}}(V_{2})={\mathcal{N}}(A)$ is obtained

2:*  if $V_{2}^{T}PV_{2}\neq 0$ then

3:*      ${\bf x}_{p}^{\bar{*}}=A^{\dagger}{\bf b}-V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})$ *

4:*      $\bar{l}^{*}=[{\bf b}^{T}(A^{\dagger})^{T}P/2+{\bf q}^{T}][I_{\it n}-V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}P]A^{\dagger}{\bf b}+s-{\bf q}^{T}V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}{\bf q}/2$ *

5:*     if “$V_{2}^{T}PV_{2}\succ 0$” and “${\bf c}_{i}^{T}{\bf x}_{p}^{\bar{*}}\leq d_{i}$ for all $i\in{\mathcal{I}}$” then

6:*        return   $\tilde{l}^{*}=\bar{l}^{*}$, ${\bf x}^{\tilde{*}}={\bf x}_{p}^{\bar{*}}$, and $\tilde{\mathcal{I}}^{*}$ *

7:*     else if “$V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})\in{\mathcal{R}}(V_{2}^{T}PV_{2})$” and “${\bf c}_{i}^{T}{\bf x}_{p}^{\bar{*}}\leq d_{i}$ for all $i\in{\mathcal{I}}~{}$” then

8:*         $(\bar{l}^{*},{\bf x}_{p}^{\bar{*}},\emptyset)\in\bar{\mathcal{L}}\times\bar{\mathcal{X}}\times\bar{\mathcal{I}}$ *

9:*         go to line 12 *

10:*     end if *

11:*  else

12:*     for all ${\mathcal{I}}_{j}\coloneqq\{j_{1},j_{2},\cdots,j_{k}\}\subseteq{\mathcal{I}}$, where $k\in\{1,2,\cdots,\kappa\}$, do

13:*         let $\tilde{C}=[{\bf c}_{j_{1}},{\bf c}_{j_{2}},\cdots,{\bf c}_{j_{k}}]^{T}$, $\tilde{A}=[A^{T},\tilde{C}^{T}]^{T}$, and $\tilde{\bf b}=[{\bf b}^{T},d_{j_{1}},d_{j_{2}},\cdots,d_{j_{k}}]^{T}$ *

14:*        if $\tilde{\bf b}\in{\mathcal{R}}(\tilde{A})$ then

15:*           if rank*$(\tilde{A})={\it m}+{\it k}<{\it n}$* then

16:*               compute the SVD of $\tilde{A}$ till ${\mathcal{R}}(\tilde{V}_{2})={\mathcal{N}}(\tilde{A})$ is obtained *

17:*              if “$\tilde{V}_{2}^{T}P\tilde{V}_{2}\neq 0$”, “$\tilde{V}_{2}^{T}({\bf q}+P\tilde{A}^{\dagger}\tilde{\bf b})\in{\mathcal{R}}(\tilde{V}_{2}^{T}P\tilde{V}_{2})$”, and “${\bf c}_{i}^{T}\tilde{\bf x}_{p}^{\bar{*}}\leq d_{i}$ for all $i\in{\mathcal{I}}\backslash{\mathcal{I}}_{j}$, where $\tilde{\bf x}_{p}^{\bar{*}}=\tilde{A}^{\dagger}\tilde{\bf b}-V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}({\bf q}+P\tilde{A}^{\dagger}\tilde{\bf b})$” then

18:*                  $\tilde{\bar{l}}^{*}=[\tilde{\bf b}^{T}(\tilde{A}^{\dagger})^{T}P/2+{\bf q}^{T}][I_{\it n}-V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}P]\tilde{A}^{\dagger}\tilde{\bf b}+s-{\bf q}^{T}V_{2}(V_{2}^{T}PV_{2})^{\dagger}V_{2}^{T}{\bf q}/2$ *

19:*                  $(\tilde{\bar{l}}^{*},\tilde{\bf x}_{p}^{\bar{*}},{\mathcal{I}}_{j})\in\bar{\mathcal{L}}\times\bar{\mathcal{X}}\times\bar{\mathcal{I}}$ *

20:*              end if *

21:*           else if rank*$(\tilde{A})={\it n}$* then

22:*               compute $\hat{\bf x}=\tilde{A}^{\dagger}\tilde{\bf b}$ *

23:*              if ${\bf c}_{i}^{T}\hat{\bf x}\leq d_{i}$, for all $i\in{\mathcal{I}}\backslash{\mathcal{I}}_{j}$ then

24:*                  $\hat{l}=F_{\bf x}(\hat{\bf x})$, where $F_{\bf x}$ is from (1) *

25:*                  $(\hat{l},\hat{\bf x},{\mathcal{I}}_{j})\in\bar{\mathcal{L}}\times\bar{\mathcal{X}}\times\bar{\mathcal{I}}$ *

26:*              end if *

27:*           end if *

28:*        end *if

29:*     end for *

30:*      $\tilde{l}^{*}=\min\bar{\mathcal{L}}$, while ${\bf x}^{\tilde{*}}$ and $\tilde{\mathcal{I}}^{*}$ are as associated *

31:*     return   $\tilde{l}^{*}$, ${\bf x}^{\tilde{*}}$, and $\tilde{\mathcal{I}}^{*}$ *

32:*  end if *

Remark 6

Regarding Algorithm 1,

1. i)

it does not require to know a feasible point at any time in the solving process (in contrast to, for example, a priori to initiate the process according to **[2, 16]**) since the core of the analytical philosophy is a novel categorization of QP Problem (15) in Appendix C, which particularly builds on new results in Lemma 1. 2. ii)

in the for-all environment (lines 12-29), the processing order is arbitrary, as long as every nonempty subset of ${\mathcal{I}}$ is examined. Examples are given later in Sec. IV. 3. iii)

an economic/expedited version of this algorithm is available with respect to Case (a) in Fig. 3, as motivated by the particular interest in the computational performance using existing QP solvers (to name a few, **[8, 23]**). Specifically, by replacing lines 8 and 9 with 6, the complete searching for all optima is economized since the optimal value and an optimum have already been obtained, without further effort for any possible, additional optimum. Accordingly, the algorithm is expedited because it terminates at line 8 without proceeding into the for-all environment (lines 12-29). More details and related discussions can be found in the end of Appendix C-A, as well as Remark 18.

Corollary 1

With regard to Algorithm 1, consider

1. A)

a special case of the QP Problem (15), $P\succ 0$ as is commonly seen in literature [16]. Algorithm 1 can be more efficiently compacted by removing lines 2, 7-9, 11, and 32, as well as “$V_{2}^{T}PV_{2}\succ 0$” at line 5, and both “$\tilde{V}_{2}^{T}P\tilde{V}_{2}\neq 0$” and “$\tilde{V}_{2}^{T}({\bf q}+P\tilde{A}^{\dagger}\tilde{\bf b})\in{\mathcal{R}}(\tilde{V}_{2}^{T}P\tilde{V}_{2})$” at line 17. 2. B)

an extension to the QP under inequality constraints only, that is, $A=0$ and ${\bf b}={\bf 0}$ in the QP Problem (15). Algorithm 1 applies to this case by replacing “both $A,A^{\dagger}$ with [math]”, “$V_{2}$* with $I_{\it n}$”, “${\bf x}_{p}^{\bar{*}}$ with ${\bf x}_{p}^{*}$”, “${\bf b}$ with ${\bf 0}$”, and “$\bar{l}^{*}$ with $l^{*}$”.*

Proof:

A): Given $P\succ 0$, in Algorithm 1 we readily have $V_{2}^{T}PV_{2}\succ 0$, $V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})\in{\mathcal{R}}(V_{2}^{T}PV_{2})={}^{{\it n}-{\it m}}$, $\tilde{V}_{2}^{T}P\tilde{V}_{2}\succ 0$, and $\tilde{V}_{2}^{T}({\bf q}+P\tilde{A}^{\dagger}\tilde{\bf b})\in{\mathcal{R}}(\tilde{V}_{2}^{T}P\tilde{V}_{2})={}^{{\it n}-{\it m}-{\it k}}$. The result immediately follows.

B): At first, the concept is to replace the application of Theorem 2 by Theorem 1 in the beginning of Algorithm 1. More specifically, the processes before the for-all environment (lines 3-10) instead consider the unconstrained QP, and determine whether the associated unique optimum, or the particular solution of nonunique optima, satisfies the (inequality) constraints. Starting from line 12, the remaining processes in Algorithm 1 are slightly modified. The main difference is that, at line 13, the remaining processes are instead built on the “base” of unconstrained QP ($A=0$ and ${\bf b}={\bf 0}$) and then taking each inequality constraint into consideration. All in all, from this novel perspective and design, this extension can be easily established by the simple parameter replacements. ∎

Remark 7

With respect to Corollary 1,

1. i)

in A) of Corollary 1, the more compacted algorithm renders ${\bf x}^{\bar{*}}={\bf x}_{p}^{\bar{*}}$ and $\tilde{\bf x}^{\bar{*}}=\tilde{\bf x}_{p}^{\bar{*}}$ for all ${\mathcal{I}}_{j}\subseteq{\mathcal{I}}$. That is, the associated solution freedom vanishes (because of $P\succ 0$). Even though Algorithm 1 is designed to implicitly while exhaustively include the optimality searching within such a freedom (more detailed arguments at Appendix C), this remark also reminds that the searching process, excessive in this case, automatically vanishes as well – as designed. 2. ii)

Example 9 in Sec. IV demonstrates both Algorithm 1 and Corollary 1.

III-D Extended QP

Theorem 3

*(Solutions to An Extended QP)

Consider the optimization problem,*

[TABLE]

where $F_{\bf x}$ is in Eq. (1).

1. A)

The preimage of any level set of the CCQF “$F_{\bf x}$* subject to both ${\bf x},~{}{\bf q}\in{\mathcal{R}}(P)$” can be respectively parameterized by*

[TABLE]

where $\check{l}\in\real$ is any level set value of the CCQF, $\check{\boldsymbol{\rho}}\in{\mathcal{R}}(P)$, and $\|\check{\boldsymbol{\rho}}\|=1$. 2. B)

The optimal value is finite, and equals $\check{l}^{*}=s-{\bf q}^{T}P^{\dagger}{\bf q}/2$. The corresponding unique optimum is ${\bf x}^{\hat{*}}=-P^{\dagger}{\bf q}$.

Proof:

See Appendix D. ∎

Corollary 2

Consider the specific but popular case, $P\succ 0$ [16]. Theorem 3 can be specialized by replacing $P^{\dagger}$ with $P^{-1}$, $P^{\dagger/2}$ with $P^{-1/2}$, and “$\check{\boldsymbol{\rho}}\in{\mathcal{R}}(P)$” with “$\check{\boldsymbol{\rho}}\in{}^{\it n}$”, respectively.

Proof:

Given $P\succ 0$, such a case-specific result readily follows by relating the basic SVD properties in [14, Eqs. (35)-(39)] to this special consideration. ∎

Remark 8

An application of Theorem 3 is associated with the CQF in Eq. (9), that is, to obtain its preimage ${\bf w}$ at any level set/value. To be more detailed, by letting $P=2M$ which is rank deficient, ${\bf q}={\bf k}_{M}$, and $s=c$ in accordance with the CQF in Eq. (1), the preimage ${\bf w}$ can be algebraically while completely parameterized by (using Theorem 3)

[TABLE]

where $\breve{F}_{\bf w}\in\real$ is any (given) level set value, $\hat{\boldsymbol{\rho}}\in{\mathcal{R}}(M)$, and $\|\hat{\boldsymbol{\rho}}\|=1$. As an example, this application-benefit is also demonstrated (with an illustration) in [14, Sec. 5].

III-E Comparisons and Impacts to Literature

With respect to the QP Problem (15), several representative and popular solvers in the literature or readily in the market [20] are selected to demonstrate this section. At first, according to [16], such a general QP (with inequality constraints) is usually, numerically solved using an “Active Set Method” (ASM), which is also valued in the fields of management science [7, 11] and control engineering [8], to name a few. It has been reported that a feasible point is required to initiate the solving process of ASM – including a simpler/common case of $P\succ 0$ [16] – till the latest MATLAB® implementation (namely, in the “quadprog” routine). Similar difficulty/inconvenience can also be found in other celebrated solvers: Newton’s method, the barrier method, and primal-dual interior-point methods [7, 2, 8]. In contrast, the proposed solvers autonomously generate the results/solutions, notably Algorithm 1 and Theorem 2. The analytical philosophy differentiates from the classical Lagrange/Primal-Dual method [2, 16, 20], but stems from Lemma 1 as well as Figs. 1 (“critical shift”) and 2 (“hierarchical layers”).

In addition, another advantage is revealed from an accuracy perspective, with regard to the other representative solver: the barrier method that is renowned for large-scale problems [8]. The importance of such an interior-point solver is particularly focused upon in [2, Chapter 11]. Accordingly, under reasonable assumptions, a minimum number of iteration steps are generally required for convergence, or a desired accuracy; whilst, the maximum number of steps (and computation time) are deemed challenging [8]. As a comparison, the proposed results yield exact solutions for both the optimal point(s) and value. Specifically, considering the equality-constrained QP Problem (10), the finite solutions (when existed, according to a comprehensive categorization in Theorem 2) are available in Eqs. (12) and (13). Moreover, regarding the general (resp., extended) QP problem in (15) (resp., (16)), Algorithm 1 (resp., Theorem 3) also renders results in analytical, algebraic, and closed forms. Remarkably, it is worth mentioning that all the proposed results refrain from any convergence analysis, namely the estimation of convergence rate, which can cause computational difficulty [5] but is generally necessary till the up-to-date literature.

Furthermore, the proposed exact optimization is beneficial in the light of its derivative-free feature. Such a property subject to special cases has a long history in literature while attracts renewed interest in recent years, for instance, [5] that highlights its necessity in the very beginning. In other words, the feature is a merit since it does not require any gradients. More specifically, it is desirable in settings in which exact gradient/first-order calculations are computationally expensive or impossible. Note that, in such cases, the classical techniques – including Kiefer-Wolfowitz-type procedures [12] – often resort to compromises that are based on approximations, which (observation) further strengthens the above mentioned advantage in accuracy while reminds a special consideration in Remark 18. Moreover, in the fields of statistics and machine learning [5, Refs. 3-7], there has been a growing spectrum of optimization problems that entail functional information only, which also broadens the impacts of this research direction. Finally, it is worth reiterating that the proposed scheme exploits all the information of the function, which (unified framework) includes both the zero-order [5] and first-order values that have been widely leveraged in the classical/iterative optimization [20, 16].

All of the above advantages will be validated in Sec. IV below, which provides explicit computational evidence. Promisingly, there exist more advantages, such as the one considered in Remark 18 (Appendix C); and those of practical importance: computational speed, hardware complexity, and safety standard, since this research direction toward an exact solution aligns with, for example, [9] and its mathematical preliminaries (notably [1], which is with respect to a positive definite Hessian matrix). Last but not least, an additional impact to the scientific literature suggests the (bi)linear matrix inequalities-based design (to name one recent representative, [19]), which traces its core value/concept to the QP problem and thus motivates an interesting further research.

IV Three Illustrative Examples

The main objective is to demonstrate the autonomy, generality, and functionality of the proposed novel closed-form QP solvers (Theorem 2, Algorithm 1, and so on), particularly from the implementation perspective. More specifically, it is worth reiterating that all the proposed results do not require any knowledge of a feasible point – at any time before and during the solving process (namely, autonomy); deal with the general QP problem in accordance with, for example, [16], as well as an extension that enlarges the application coverage (generality); and exploit all the information of the function that includes the commonly utilized zero-order [5] and first-order functional values in the classical/numerical optimization [20] (functionality). Example 9 is directly adopted from [16], and used to demonstrate Theorems 1, 2, Algorithm 1, Corollary 1, Remarks 1, 3, 7, 17, and Lemma 1. One step forward, Example 10 is extended and modified from Example 9, by adding an equality constraint, rendering the Hessian matrix singular, and increasing the system dimension. Therefore, it also demonstrates those mentioned above, except the special cases of Corollary 1 and Remark 7, and additionally Remark 15 for the concept of terminal optima and related solution freedom. For compactness, the illustration for Example 9 can be either inferred from Fig. 4 or referenced to [16, 2nd edn., Fig. 14.1]; whereas, in Example 10, a geometric interpretation for Remark 15 is given in Fig. 5. Last but not least, Example 11 leverages a benchmark problem as selected in [18] and compares the proposed approach with the built-in QP solver “quadprog”, which (among others) boasts a wide popularity in the optimization literature and beyond [25]. Note that all computations in the three examples are performed and/or verified on the MATLAB® platform, and they are subject to a variety of hardware implementations (details in Table I later).

Example 1

Consider a QP problem of nonsingular Hessian matrix and under inequality constraints only, where ${\it n}=2$, $P(1,:)=[4,1]$, $P(2,:)=[1,2]$, ${\bf q}=[-12,-10]^{T}$, $s=0$, ${\bf c}_{1}=[1,1]^{T}$, ${\bf c}_{2}=[-1,0]^{T}$, ${\bf c}_{3}=[0,-1]^{T}$, $d_{1}=4$, and $d_{2}=d_{3}=0$ [16]. Given $P\succ 0$, according to Corollary 1, iii) of Theorem 1, or line 3 of Algorithm 1, we have the unique optimum with respect to the unconstrained problem as ${\bf x}^{*}=-P^{-1}{\bf q}=[2,4]^{T}$, which does not satisfy the inequality constraint associated with $({\bf c}_{1},d_{1})$. That is, this unconstrained optimum does not reside in the feasible set. Therefore, following Algorithm 1, the next step is the for-all environment starting at line 12, where ${\mathcal{I}}=\{1,2,3\}$ ($\kappa=3$). Note that the significant parameters and values in this step are summarized by Table II in Appendix E for completeness and easy comparison. Moreover, the examination order for all ${\mathcal{I}}_{j}$’s is arbitrary, as long as every nonempty subset of ${\mathcal{I}}$ is examined, which is discussed in ii) of Remark 6.

In Table II (Appendix E), at first we note that only those ${\mathcal{I}}_{j}$’s with $\tilde{\bf b}\in{\mathcal{R}}(\tilde{A})$ in the corresponding augmented system are included, as filtered at line 14 in Algorithm 1; otherwise, it does not associate with the optimality since being inconsistent, and the only case is ${\mathcal{I}}_{7}$, where, equivalently,

[TABLE]

Second, regarding ${\mathcal{I}}_{1}$, the corresponding augmented matrix $\tilde{A}$ is of full rank, and the augmented system $\tilde{A}{\bf x}=\tilde{b}$ is underdetermined and has more than one solution. This falls into the processes at lines 16-20 of Algorithm 1. According to Theorem 2, an example of $\tilde{V}_{2}=[1/\sqrt{2},-1/\sqrt{2}]^{T}$ is by means of SVD, which is computed at line 16 of Algorithm 1. This yields the value “$\tilde{V}_{2}P\tilde{V}_{2}=2\neq 0$”, which implies that this case is equivalently an unconstrained QP. As a matter of fact, given $P\succ 0$, the following results also endorse Corollary 1:

1. $\bullet$

$\tilde{V}_{2}^{T}({\bf q}+P\tilde{A}^{\dagger}\tilde{b})\in{\mathcal{R}}(\tilde{V}_{2}P\tilde{V}_{2})={\mathcal{R}}(2)=\real$, which means that the equivalent QP problem is of finite value by A)c) of Theorem 2. 2. $\bullet$

${\mathcal{N}}(P)=\{{\bf 0}\}$, and thus ${\mathcal{N}}(\tilde{A})\cap{\mathcal{N}}(P)=\{{\bf 0}\}$. According to A)e) of Theorem 2 or i) of Remark 7, we have that the associated optimum for this case ${\mathcal{I}}_{1}$ is unique and denoted $\tilde{\bf x}^{\bar{*}}=\tilde{\bf x}_{p}^{\bar{*}}=[1.5,2.5]^{T}$. This point is computed at line 17 in Algorithm 1. 3. $\bullet$

$\tilde{\bf x}^{\bar{*}}$* is an optimality candidate for the original problem, as recognized at line 19, since it satisfies the other (inequality) constraints with respect to $\{({\bf c}_{i},d_{i})\}_{i=2,3}$. In other words, it is (also) a feasible point with regard to the original problem/constraints.* 4. $\bullet$

The associated optimal value ($\tilde{\bar{l}}^{*}=-28.5$) for this case ${\mathcal{I}}_{1}$ is determined by Eq. (12), computed at line 18, and included at line 25 for the overall comparison later at line 30 of Algorithm 1.

Similar considerations apply to Cases $\{{\mathcal{I}}_{j}\}_{j=2,3}$, but note that the associated unique optimum $\left(\tilde{\bf x}^{\bar{*}}=[0,5]^{T}\right)$ for Case ${\mathcal{I}}_{2}$ is not a candidate, because it violates the first inequality constraint “$x_{1}+x_{2}\leq 4$”.

Third, Case ${\mathcal{I}}_{4}$ corresponds to a “vertex”, which is zero-dimensional, as a result of rank$(\tilde{A})={\it n}=2$. Specifically, according to ii) of Remark 3, this case is instead included at lines 21-27 of Algorithm 1, and there is only one feasible point $\hat{\bf x}=[0,4]^{T}$ as computed at line 22. Since $\hat{\bf x}$ satisfies all the constraints, it is recognized as a candidate (line 25). The associated, only level set value $\hat{l}=-24$ is exclusively optimal in this case, computed at line 24, and thus included for the comparison at line 30. Similar considerations apply to Cases $\{{\mathcal{I}}_{j}\}_{j=5,6}$, but omitted for conciseness.

Finally, all the values, either $\tilde{\bar{l}}^{*}$ or $\hat{l}$, are compared at line 30. This concludes that Case ${\mathcal{I}}_{1}$ corresponds to the optimality. To be more detailed, ${\bf x}^{\tilde{*}}=\tilde{\bf x}^{\bar{*}}=[1.5,2.5]^{T}$ for ${\mathcal{I}}_{1}$ is also the unique optimum for the original problem, with the optimal value $\tilde{l}^{*}=\tilde{\bar{l}}^{*}=-28.5$, and ${\bf x}^{\tilde{*}}$ is located on the edge of ${\mathcal{F}}$ that is associated with ${\mathcal{I}}_{1}$, explicitly, $x_{1}+x_{2}=4$.

Remark 9

The results agree with [16] that uses ASM, but without any prior knowledge of a feasible point to initiate the solving process. Besides, all computations are performed in terms of analytical, algebraic, closed-form representations. Moreover, there are only seven cases considered in the for-all environment of Algorithm 1 (lines 12-29). In consistency with Remark 17, the cardinality of $\bar{\mathcal{L}}$, which equals 5, is less than the upper bound “$2^{\kappa}=8$”.

Example 2

This example is extended and modified from Example 9. The main difference in the motivation is additionally to demonstrate the concept of terminal optima, as explained in Appendix C-B, and a potential to assist the analytical and comprehensive representation of all the nonunique optima. According to the problem formulation in QP (15), we let ${\it n}=3$, singular $P=\mbox{diag}(1,0,0)\succeq 0$, ${\bf q}={\bf 0}$, $A=[0~{}0~{}1]$, ${\bf b}=b\in\real$, while the other parameters follow Example 9. For succinctness, only the significantly different analyses and results are presented below.

First (lines 1-10 in Algorithm 1), consider the QP under only the equality constraint ($A{\bf x}=b$). At line 1, $V_{2}$ is computed as $V_{2}(:,1)=[0~{}1~{}0]^{T}$ and $V_{2}(:,2)=[1~{}0~{}0]^{T}$. The associated matrix $V_{2}^{T}PV_{2}=\mbox{diag}(0,1)\neq 0$, which implies that this is equivalently an unconstrained QP by A) of Theorem 2. In addition, given the singular $V_{2}^{T}PV_{2}\succeq 0$ as well as $V_{2}^{T}({\bf q}+PA^{\dagger}{\bf b})={\bf 0}\in{\mathcal{R}}(V_{2}^{T}PV_{2})$, by A)c) of Theorem 2, the optimal value (that is, minimum) is finite. Moreover, since ${\mathcal{N}}(A)\cap{\mathcal{N}}(P)={\mathcal{R}}([0~{}1~{}0]^{T})\neq\{{\bf 0}\}$, applying A)e) of Theorem 2 yields that the optima are nonunique. Preliminarily at this stage, Algorithm 1 only examines the particular solution of the optima (${\bf x}_{p}^{\bar{*}}$, as computed at line 3), while integrates the examination on its solution freedom ($V_{2}{\boldsymbol{\varepsilon}}^{\bar{*}}$) into the for-all environment later (lines 12-29). The result is ${\bf x}_{p}^{\bar{*}}=[0,0,b]^{T}$, which resides in the feasible set ${\mathcal{F}}$ of the original problem (grey/shaded area in Fig. 5). Hence it qualifies as an optimality candidate (included at line 8), associated with $\bar{l}^{*}=0$ (computed at line 4).

Second, the process goes to the for-all environment in Algorithm 1. Following the analyses in Example 9, the results are briefed and summarized by Table III in Appendix E for better presentation clarity. Note that:

1. $\bullet$

Considering ${\mathcal{I}}_{1}$, this corresponds to the terminal optimum associated with the above mentioned solution freedom of ${\bf x}^{\bar{*}}$ (namely, $V_{2}{\boldsymbol{\varepsilon}}^{\bar{*}}$) as a result of the additional constraint with respect to $({\bf c}_{1},d_{1})$. This can also be inferred from Fig. 5, where the red/solid/vertical lines represent the level sets/values of the involved CQF; while the $x_{2}$-axis is just the linear variety ${\bf x}^{\bar{*}}$, and the terminal optima $(0,4,b)$ is owing to the additional constraint $({\bf c}_{1},d_{1})$. This demonstration agrees with the discussions in Appendix C-A. 2. $\bullet$

Regarding ${\mathcal{I}}_{2}$, this is equivalently an optimization problem but of constant value, according to B) and C) of Theorem 2. Specifically, it is owing to $\tilde{V}_{2}^{T}P\tilde{V}_{2}=0$, $\tilde{V}_{2}^{T}({\bf q}+P\tilde{A}^{\dagger}\tilde{\bf b})=0$, and the constant value equals $s+[{\bf q}^{T}+\tilde{\bf b}^{T}(\tilde{A}^{\dagger})^{T}P/2]\tilde{A}^{\dagger}\tilde{\bf b}=0$. Additionally, as addressed in ii)b) of Appendix C-B, the algorithm is designed to economize this examination by focusing on the associated terminal optima. Specifically, it is instead included in Cases $\{{\mathcal{I}}_{j}\}_{j=4,5}$, and associated with additional constraints $\{({\bf c}_{i},d_{i})\}_{i=1,3}$, respectively. Notably, the associated optimal value remains. Finally, a geometric interpretation is also available in Fig. 5, where, on the $x_{2}$-axis (that is, the border of ${\bf c}_{2}^{T}{\bf x}\leq d_{2}$), the equivalent problem/function bears the constant value. In its domain, the upper (resp., lower) terminal is due to the additional constraint “$({\bf c}_{1},d_{1})$” (resp., “$({\bf c}_{3},d_{3})$”), and the discussion/categorization goes to the case ${\mathcal{I}}_{4}$ (resp., ${\mathcal{I}}_{5}$). 3. $\bullet$

Noting the additional column of ${\mathcal{N}}(P)\cap{\mathcal{N}}(\tilde{A})$ in Table III, which is omitted in Table II (Appendix E) because of the more convenient condition “$P\succ 0$” instead/previously in Example 9, the case ${\mathcal{I}}_{1}$ (resp., ${\mathcal{I}}_{3}$) renders this intersection only at the origin, and thus the corresponding optimum is unique, $\tilde{\bf x}_{p}^{\bar{*}}=\tilde{\bf x}^{\bar{*}}$, according to A)e) of Theorem 2. This can also be inferred from Fig. 5, where there is only one intersection between the optimal (level set) value, denoted $\tilde{l}^{*}$, and the border of ${\bf c}_{1}^{T}{\bf x}\leq d_{1}$ (resp., ${\bf c}_{3}^{T}{\bf x}\leq d_{3}$).

Finally, corresponding to the optimal value $\tilde{l}^{*}$, at line 30 in this algorithm, it concludes with the five optimality pairs/points residing in $\bar{\mathcal{L}}\times\bar{\mathcal{X}}\times\bar{\mathcal{I}}$: $([0,0,b]^{T},0,\emptyset)$, $([0,4,b]^{T},0,{\mathcal{I}}_{1})$, $([0,0,b]^{T},0,{\mathcal{I}}_{3})$, $([0,4,b]^{T},0,{\mathcal{I}}_{4})$, and $([0,0,b]^{T},0,{\mathcal{I}}_{5})$. That is, Algorithm 1 gives the optimal value, $\tilde{l}^{*}=0$, with respect to the original problem. All the terminal optima are $[0,0,b]^{T}$ and $[0,4,b]^{T}$, as illustrated in Fig. 5. Notably, the former i) is the particular solution with regard to the QP under only the equality constraint, and also feasible to the original QP; and ii) resides on an edge associated with the inequality constraint “${\bf c}_{3}^{T}{\bf x}\leq d_{3}$”, while also on a vertex with the additional constraint “${\bf c}_{2}^{T}{\bf x}\leq d_{2}$”. On the other hand, the latter terminal optimum resides on an edge associated with the constraint “${\bf c}_{1}^{T}{\bf x}\leq d_{1}$”, while also on a vertex with the additional “${\bf c}_{2}^{T}{\bf x}\leq d_{2}$”. It is worth remarking that Algorithm 1 ably gives all/both the terminal optima. Based on this, it can be analyzed that all the optima are just intermediate, denoted as ${\mathcal{X}}^{*}=\{[0,t,b]^{T}~{}\vrule~{}0\leq t\leq 4,~{}t\in\real\}$, which can also be revealed from Fig. 5 (specifically, the red-dashed line segment).

Remark 10

As expected and similar to Example 9 (Remark 9), Algorithm 1 autonomously yields the results, with only functional information (derivative-free) and without any (prior) knowledge of a feasible (starting) point. Also, in this setting, all computations are performed in terms of algebraic and explicit representations. Moreover, there are still only few cases (seven, to be exact) considered in the for-all environment of Algorithm 1 and, being consistent with Remark 17, the cardinality of $\bar{\mathcal{L}}$ (6) is indeed less than the upper bound “$2^{\kappa}=8$”.

Example 3

The extensive QP literature highlights a universal and popular solver “quadprog” in MATLAB® [25], and this example leverages a benchmark problem as demonstrated in [18] to offer more computational evidences of the superiority of the proposed approach. Specifically, let $P(1,:)=[1,-1,1]$, $P(2,:)=[-1,2,-2]$, $P(3,:)=[1,-2,4]$, ${\bf q}=[-7,-12,-15]^{T}$, $s=0$, $A=[1,1,1]$, and $b=3$ in Problem/Eq. (10), and this Example 11 compares Theorem 2 with “quadprog”. Among various implementations for Theorem 2, we demonstrate an intuitive and preliminary one using MATLAB® for the sake of brevity, whose primary steps are sequentially summarized:

⬇

1A_dagger**=pinv(A);

2V_2**=null(A);

3x_star_particular**=A_daggerb…*

4* -V_2pinv**(V_2’PV_2)(V_2’q+V_2’PA_daggerb);*

5l_star_equilibrium**=(b’A_dagger’P/2+q’)(eye(n)-V_2pinv(V_2’PV_2)…

6* V_2’*P)A_daggerb*+s-q’V_2pinv(V_2’PV_2)V_2’q/2; *(i)

Note that lines 3 and 4 in Implementation (i) correspond to Eq. (13) while 5 and 6 to (12). In addition, this example falls into Case A) of Theorem 2 because of ${\mathcal{R}}(P)\cap{\mathcal{N}}(A)={\mathcal{N}}(A)\neq\{{\bf 0}\}$, which can be easily verified – using $V_{2}^{T}PV_{2}\neq 0$ – and thus omitted in the summary of primary steps (i).

On the MATLAB® platform, the proposed solver yields the optimal value $\bar{l}^{*}=-47.1786$ and the optimum ${\bf x}^{\bar{*}}=[-3.5714,2.9286,3.6429]^{T}$, where the uniqueness of $\bar{l}^{*}$ (resp., ${\bf x}^{\bar{*}}$) is owing to the nonsingular $V_{2}^{T}PV_{2}$, namely the nulled flexibility ${\boldsymbol{\varepsilon}}^{\bar{*}}={\bf 0}$ in Eq. (13); whereas, the built-in “quadprog” routine renders the same $\bar{l}^{*}$ and ${\bf x}^{\bar{*}}$ but not the associated uniqueness, respectively [18]. Moreover, the performance in computation time also quite differ, which is evaluated – for better comprehensiveness – subject to a variety of software and hardware in Table I. Specifically, Table I also more favors Theorem 2 than “quadprog”, where the evaluation is in terms of the ratio of computation time (Theorem 2 over “quadprog”) while among several Intel Core® CPUs and MATLAB® versions till the state of the art (with 8GB RAM). Accordingly, the maximum (resp., minimum) advantage amounts to 2.3$\%$ (resp., 9.6$\%$) of the computation time using “quadprog”; in other words, the proposed solver reduces the computational effort by 90.4$\%-$97.7$\%$. In average, the numerics still significantly account for a computational reduction of 94.5$\%$ by virtue of Theorem 2. It is worth reiterating that the Implementation (i) is only preliminary, whose efficiency potentials can be more exploited by, for instance, “coalescing” lines 1 and 2, namely, by performing the SVD operation on $A$ only once. This requires modifications for the “svd” routine in MATLAB®, and thus serves as a future research to remain focused of this article.

Remark 11

Regarding the up-to-date universal solver “quadprog”, [18] highlights a necessary condition “$P\succ 0$” that associates with a finite optimal value, even if the QP problem is without any constraint (Sec. III-A). Actually, there exists a counterexample to the condition/necessity: $P=\mbox{diag}(1,0)$, ${\bf q}={\bf 0}$, and $s=0$ in Eq. (1). That is, the QP problem is to minimize $F_{\bf x}({\bf x})=x_{1}^{2}$, which has a/the finite minimum (zero) whereas the Hessian matrix is not positive definite. As a comparison, ii) of Theorem 1 provides a necessary and sufficient condition, which readily guarantees the finite minimum in the above counterexample – owing to ${\bf q}={\bf 0}\in{\mathcal{R}}(P)$. Remarkably, Theorem 1 fundamentally helps develop the remaining results (with constraints) in Sec. III, such as another equivalent condition for the finite optimal value in A)c) of Theorem 2.

V Conclusion

A motivation of this application – to the field of nonlinear programming based on the recent observations of CQE – responds to an open question in [5]; and an expectation in the general optimization literature: computational enhancement for practical advances [16]. The philosophy aims at analytical and exact optimization, as a comparison to the common numerical algorithms [27, 20]. Specifically, the focus of this article is on the QP, which is a constituent in various approaches such as the Newton’s method. At first, we re-examine the unconstrained QP based on, and from the perspective of, the new analyses of CQE (in the recent literature), CQF, and CQE-CQF relation. This preliminary finding facilitates a complete and analytical characterization of the equality-constrained QP, which actually can be categorized into three equivalent problems in a unified framework. In addition, all the above results are consistent with the literature, when specialized to specific considerations; while promising in terms of the computational performance, which is partially owing to the shared motivation and beneficial results in the recent contribution [14]. Another highlight in this presentation is the proposed QP solver, in accordance with the general problem formulation. The analytical philosophy also benefits from the above mentioned categorization of equality-constrained QP, and yields algebraic zero-order closed-form results/solutions in a guaranteed, pre-determined, finite, and explicit number of steps; and without any knowledge of a feasible point, a priori and any time in the process. All these characteristics/merits are shared throughout this article. Moreover, to further exploit its computational capabilities, this flexible QP solver includes problem-specific (and more efficient) variants below:

1. $\bullet$

an economic/expedited version for a subset of QP problems; 2. $\bullet$

a special but popular case “the objective function whose Hessian matrix is positive definite”, which corresponds to a thinned version of the algorithm; 3. $\bullet$

QP problems subject to inequality constraints only, being applicable (that is, solvable) after a simple replacement of parameters in the algorithm.

Note that, regarding a subset of QP problems, we preliminarily and explicitly present the associated terminal optimum/optima. This indicates a further research direction, because it requires an analytical representation of all the elements bounded by the inequality constraints. Moreover, we extend the general formulation on the constraints to a different branch, as inspired by the analyses of CQE and CQF, which further broadens the spectrum of QP applications. Last but not least, two representative solvers in literature (the ASM and barrier method) help justify advantages/potentials of the proposed results; while three examples demonstrate the results with illustrations, computational evidences, and further insights (notably, a counterexample to the state of the art) – analytically as well as numerically on the MATLAB® platform. Under positive definite and semidefinite Hessian matrices in the objective functions, respectively, these demonstrations endorse the effectiveness, autonomy, efficiency, functionality, and exactness of the proposed solvers.

Appendix A Proof of Theorem 1 (Unconstrained QP)

For brevity, only the results that are significantly required in this proof are given, whereas most of the shared similarities are referenced to Lemma 1 and its proof.

i) To obtain the preimage of any level set, $F_{\bf x}=l$, is equivalent to that of the zero level set of $\tilde{F}_{\bf x}$, where the CQF $\tilde{F}_{\bf x}:{}^{\it n}\rightarrow\real$ and $\tilde{F}_{\bf x}=F_{\bf x}-l$, as “(vertically) shifted” from $F_{\bf x}$ by $l$. Furthermore, this is equivalent to solving the CQE, ${\bf x}^{T}P{\bf x}+{\bf q}^{T}{\bf x}+s-l=0$, which is always solvable since the original process is to find the preimage. The result directly follows by applying Lemma 1 to this CQE. Note that the above mentioned “shift” is widely used throughout Sec. III, and thus exemplified/illustrated in Fig. 1 for presentation clarity.

ii) Divide the derivations, following Lemma 1, into whether $P$ is of full rank. A) If rank$(P)={\it n}$, then we reformulate the CQF $F_{\bf x}$ in Eq. (1) equivalently as

[TABLE]

which is similar to [14, Eq. (34)]. Obviously, this function (27) is bounded below by the value “$(s-{\bf q}^{T}P^{-1}{\bf q}/2)$”, which (value) is finite and important in the derivations afterward. Note that, in this case, ${\bf q}\in{\mathcal{R}}(P)={}^{\it n}$. On the other hand, B) if rank$(P)=r<{\it n}$, then reformulate $F_{\bf x}$ (1) equivalently as

[TABLE]

which is similar to [14, Eq. (42)]. Obviously, if ${\bf q}_{2}={\bf 0}$, that is, ${\bf q}\in{\mathcal{R}}(P)$, then the domain of $F_{\bf x}$ shrinks and corresponds to the ${\bf x}_{1}$-freedom in ${\bf x}$. Similar to A) above, $F_{\bf x}$, or equivalently the CQF $F_{{\bf x}_{1}}:{}^{\it r}\rightarrow\real$,

[TABLE]

is bounded below by $\breve{l}\coloneqq s-{\bf q}_{1}^{T}\breve{\Sigma}_{1}{\bf q}_{1}/2=s-{\bf q}^{T}P^{\dagger}{\bf q}/2\in\real$. However, if ${\bf q}_{2}\neq{\bf 0}\Leftrightarrow{\bf q}\not\in{\mathcal{R}}(P)$, then $F_{\bf x}$ in Eq. (28) is unbounded because of the ${\bf x}_{2}$-freedom, which is decoupled from ${\bf x}$.

iii) Given ${\bf q}\in{\mathcal{R}}(P)$, the derivations differ only in the singularity of $P$ according to Conditions/Eqs. (3) and (5), which are divided into Cases A) and B) below.

A) If rank$(P)={\it n}$, by i) above and Eq. (4), the unique optimum happens exclusively when the only solution freedom (${\bf v}$) is canceled. In other words, the term in the square-root operator (“$-l^{*}$” as in this theorem) vanishes. Geometrically from Fig. 1, this can also be explained by the critical shift of $l^{*}$, which allows the only one-point intersection with the zero plane/line. It is worth mentioning that this shift can be “ascending” (resp., “descending”), if $l^{*}<0$ (resp., $l^{*}>0$), that is, the critical “distance” to null the square-root operation (resp., make it consistent). For succinctness, another explanation using Eq. (27) is omitted. To summarize, the preimage of the level set at the optimal value $l^{*}$ is the singleton $\{{\bf x}^{*}\}$.

B) If rank$(P)=r<{\it n}$, we at first introduce the concept of hierarchical layers in the preimage of CQF in Eq. (1), according to i) above and Eq. (7). There are three layers: I) the unique point, ${\bf x}_{p}^{*}=-P^{\dagger}{\bf q}$, II) the ${\boldsymbol{\varepsilon}}$-freedom in ${\mathcal{N}}(P)$, and III) the ${\boldsymbol{\rho}}$-freedom in ${\mathcal{R}}(P)$. This I)-III) order follows the size of included preimage elements, and I) being the smallest. Geometrically, the I) and II) layers correspond to the optima, where II) mainly reflects the singularity of $P$. Then, introducing III) equips with the full freedom, which thus corresponds to the complete preimage. For brevity, other explanations by means of, for instance, a) the geometrical perspective, which interprets the critical shift of optimal value $l^{*}$ similarly as in A); b) Eq. (29), which can be reformulated similar to (27); and c) the support from [14, Theorem  3.2] (and its proof) are omitted.

Remark 12

As an example, Fig. 2 illustrates the concept of “hierarchical layers”. Regarding the singular $P$, the three layers are represented by the I) black/crossed point: this particular solution is denoted by ${\bf x}_{p}^{*}=-P^{-1}{\bf q}$ at the center of the figure, II) blue/gridded closed region, and III) red/dashed region, respectively. All the layers are within the preimage of $F_{\bf x}$, denoted by $F_{\bf x}^{-1}\coloneqq F_{\bf x}^{-1}(l):\real\rightarrow{}^{\it n}$ with respect to any level set value $l$. I) and II) layers are in the same level set at (the optimal value) $l^{*}$, while each red/dashed ellipse in the III) layer corresponds to a level set at a higher value. Moreover, II) corresponds to ${\mathcal{N}}(P)$ and each element/point is referenced by the vector $\breve{\boldsymbol{\epsilon}}$; while III) to a point $\Phi\cdot\breve{\boldsymbol{\rho}}\in{\mathcal{R}}(P)$, where $\Phi\coloneqq\Phi(l)=\sqrt{2l+{\bf q}^{T}P^{\dagger}{\bf q}-2s}\cdot P^{\dagger/2}\in{}^{{\it n}\times{\it n}}$ while $\breve{\boldsymbol{\rho}}\in{\mathcal{R}}(P)\subseteq{}^{\it n}$ is of unit length. Note that ${\mathcal{R}}(\Phi)={\mathcal{R}}(P^{\dagger/2})={\mathcal{R}}(P)$ and $\|\Phi\|=\sqrt{(2l+{\bf q}^{T}P^{\dagger}{\bf q}-2s)/\sigma_{\it r}}$ [6], where $\sigma_{\it r}>0$ denotes the smallest (positive) singular value of $P$. On the other hand, considering the case of invertible $P$, the only differences are

1. $\bullet$

the lack of II) layer, because of ${\boldsymbol{\varepsilon}}\in{\mathcal{N}}(P)=\{{\bf 0}\}$ and $P^{\dagger}=P^{-1}$; 2. $\bullet$

$\Phi=\sqrt{2l+{\bf q}^{T}P^{-1}{\bf q}-2s}\cdot P^{-1/2}$* and $\|\Phi\|=\sqrt{(2l+{\bf q}^{T}P^{-1}{\bf q}-2s)/\sigma_{\it n}}$, where $\sigma_{\it n}$ denotes the smallest (positive) singular value or, equivalently here, eigenvalue of $P$.*

Finally, this concept of “hierarchical layers” is essential for the subsequent derivations and results in Sec. III, particularly the constrained optimization as analytically solved from a novel perspective (Secs. III-B and III-C).

Appendix B Proof of Theorem 2 (Equality Constraints)

Let the SVD of $A$ be given by

[TABLE]

with its unique pseudoinverse $A^{\dagger}\in{}^{{\it n}\times{\it m}}$ [6],

[TABLE]

where $W\in{}^{{\it m}\times{\it m}}$, $V_{1}\in{}^{{\it n}\times{\it m}}$, $V_{2}\in{}^{{\it n}\times({\it n}-{\it m})}$, ${\mathcal{R}}(V_{1})={\mathcal{R}}(A^{T})$, ${\mathcal{R}}(V_{2})={\mathcal{N}}(A)$, $\Gamma_{1}\in{}^{{\it m}\times{\it m}}$ is nonsingular, and more properties are widely available in literature [6]. The idea is to equivalently transform the equality-constrained QP into an unconstrained optimization problem, by down-casting to the linear variety/constraint as described by $A{\bf x}={\bf b}$. Specifically, the first step is to parameterize all the feasible points, namely $A{\bf x}={\bf b}\Leftrightarrow$ Eq. (11), where ${\bf y}\in{}^{{\it n}-{\it m}}$. Then, reformulate the considered constrained optimization problem, “minimize $F_{\bf x}$ (1) subject to $A{\bf x}={\bf b}$”, equivalently as the following one: minimize $F_{\bf y}$ with respect to ${\bf y}$, where $F_{\bf y}:{}^{{\it n}-{\it m}}\rightarrow\real$,

[TABLE]

Therefore, the QP Problem (10) is equivalently categorized into the three cases A)-C), according to the coefficients of quadratic, linear, and constant terms in (38), respectively. The system dimension is expectedly reduced, from n to n-m. The following derivations focus on the case of (equivalently unconstrained) QP, while it is straightforward to derive the other cases and thus omitted for compactness.

To start with, we further analyze the equivalence conditions such that $F_{\bf y}$ in Eq. (38) is quadratic, intentionally/preferably in terms of the original, given parameters. Let the SVD of (non)singular $P\succeq 0$ be

[TABLE]

where $\check{U}_{1}\in{}^{{\it n}\times{\check{\it r}}}$ is with orthonormal column(s), $\check{\it r}\coloneqq\mbox{rank}(P)$, and $\check{\Sigma}_{1}\in{}^{{\check{\it r}}\times{\check{\it r}}}$ is nonsingular. With respect to the case of singular $P$, more properties and details regarding this SVD similarly follow [14, Eqs. (35)-(37)], which can be easily extended to the nonsingular/other case. It is worth emphasizing that the following derivations coalesce both cases of $P$. On the one hand, this is a QP, namely, $F_{\bf y}$ in Eq. (38) is a CQF, iff $V_{2}^{T}PV_{2}\neq 0\Leftrightarrow V_{2}^{T}\check{U}_{1}\check{\Sigma}_{1}\check{U}_{1}^{T}V_{2}\neq 0\Leftrightarrow\check{U}_{1}^{T}V_{2}\neq 0\Leftrightarrow{\mathcal{R}}(P)\cap{\mathcal{N}}(A)\neq\{{\bf 0}\}$. Similarly on the other, this is not a QP, iff $\check{U}_{1}^{T}V_{2}=0\Leftrightarrow\mbox{``}{\mathcal{N}}(A)\subseteq({\mathcal{R}}(P))^{\perp}={\mathcal{N}}(P)$”, where the equality holds when ${\mathcal{R}}(P)\oplus{\mathcal{N}}(A)={}^{\it n}$.

Notably, to conform to the applicable form using Theorem 1, any level set of $F_{\bf y}$ (38) is regarded as a CQE (2) in the unknown variable ${\bf z}={\bf y}$, with the parameters given in A)b) of this theorem. Applying Theorem 1 yields the results in A), except A)e), while omitting the lengthy but somewhat straightforward calculations. Regarding the exceptional/last one A)e), based on the above derivations, the optimum is unique, iff the Hessian matrix of the CQF $F_{\bf y}$ in Eq. (38) is positive definite. Rewrite this equivalence condition as $V_{2}^{T}PV_{2}=V_{2}^{T}\check{U}_{1}\check{\Sigma}_{1}\check{U}_{1}^{T}V_{2}\succ 0$, according to the thin version of Eq. (39). Note that the full version corresponds to the case of nonsingular $P$, and the result similarly follows. Let any $\tilde{\bf y}\in{}^{{\it n}-{\it m}}\backslash\{{\bf 0}\}$ be given, this condition is further equivalent to

[TABLE]

where $\check{\Sigma}_{1}^{1/2}\coloneqq\mbox{diag}(\sqrt{\sigma_{1}},\sqrt{\sigma_{2}},\cdots,\sqrt{\sigma_{\check{\it r}}})\succ 0$ and $\{\sigma_{\it i}\}_{{\it i}=1}^{\check{\it r}}$ denotes the set of nonzero singular values or, equivalently in this case, eigenvalues of $P$.

Remark 13

Regarding the SVD of $A$, the columns of $V_{2}$ are not unique. However, the effect by this nonuniqueness is completely coalesced by virtue of Eq. (11), in the very beginning of this proof. In other words, the results in Theorem 2 are independent of the nonunique choices of $V_{2}$. From the implementation perspective, in the literature there exist many and various algorithms to construct an example, such as the Golub-Reinsch SVD till the second step [6]. Note that, in the MATLAB® platform, the “null” function ably gives an example of $V_{2}\in{\mathcal{N}}(A)$, which is implemented by computing the SVD of $A$.

Remark 14

For computational purposes, we select the SVD process to obtain a basis of ${\mathcal{N}}(A)$, that is, the right singular vectors of $A$ that are associated with the zero singular value. As expected, the derivations only utilize the property of ${\mathcal{R}}(V_{2})={\mathcal{N}}(A)$ where $V_{2}\in{}^{{\it n}\times({\it n}-{\it m})}$, and can be easily extended to the general case: any matrix in n×(n-m) with its range spanning ${\mathcal{N}}(A)$. However, owing to many computational benefits [6] while noting the largely used parameter $A^{\dagger}$ (and $P^{\dagger}$) in Sec. III, the SVD process is more reasonably preferred to construct an example. Besides, regarding the extension in Sec. III-C, the proposed algorithm is based on this theorem, and thus shares such a consideration.

Appendix C Proof of Algorithm 1 (Exact QP Solver)

Outline: The proof builds on the concept “hierarchical layers” as illustrated in Fig. 2, which clarifies the geometric structure of the solution to CQE (2) that is associated with the optimality/minimality of CQF (1); and case-by-case arguments in accordance with Fig. 3, which exactly extract the optimality for most of the cases, except a subset that requires further investigation (parameterization of inequality-constrained feasible points) to complementarily locate all the optima. Moreover, the derivations largely utilize Theorem 2, which are divided into: Appendix C-A corresponding to lines 2-10 in Algorithm 1; and C-B to lines 11-30, which augments/coalesces the inequality constraints with respect to each nonempty subset (line 12). Note that the design of categorizations examines every “optimality candidate” (a demonstration in Fig. 4) for the ultimate comparison at line 30, and benefits efficient extensions to QP variants such as those in Corollary 1.

To start with, Fig. 3 illustrates the main categorization for the derivations in this proof. With regard to the QP Problem (15) under only equality constraints, (a) considers the case that the unique optimum, or the particular solution ${\bf x}_{p}^{\bar{*}}$ associated with the optima, resides in the feasible set ${\mathcal{F}}$ (shaded/grey area, a polyhedron [2, 16]). If the considered point in (a) is outside of ${\mathcal{F}}$, but the degree of freedom in (13) associated with the case of nonunique optima ($V_{2}{\boldsymbol{\varepsilon}}^{\bar{*}}$) intersects ${\mathcal{F}}$, then this is included in (b). On the other hand (${\bf x}^{\bar{*}}\not\in{\mathcal{F}}$), the derivations go to either (c) or (d), where (c) addresses the case that the optimality occurs on an “edge/space of nonzero dimension” while (d) at a “vertex/point”. Note that ${\mathcal{F}}\neq\emptyset$, according to the QP problem formulation in (15). Moreover, denote $\tilde{\mathcal{I}}^{*}\subseteq{\mathcal{I}}$ as the optimality subset of involved inequality constraints, which associates with the additional equality constraint(s) that is/are the border(s) of involved inequality constraint(s). This subset indicates the “edge(s)” or “vertex/vertices” that the optimum/optima reside(s). A special case is $\tilde{\mathcal{I}}^{*}=\emptyset$, which corresponds to the case when QP Problems (10) and (15) yield the identical solution. Finally, the first line (fundamental step) of this algorithm is to compute an orthonormal basis of ${\mathcal{N}}(A)$, denoted $V_{2}$, preferably using SVD due to its many computational advantages [6].

C-A Case (a) ${\bf x}_{p}^{\bar{*}}\in{\mathcal{F}}$

According to Theorem 2, this case corresponds to A) where $V_{2}^{T}PV_{2}\neq 0$, as filtered at line 2. Note that the case of $V_{2}^{T}PV_{2}=0$ corresponds to either an LP or constant function. If, due to additional inequality constraints, the optimality of QP Problem (15) is associated with the former (resp., latter), then the optimality occurs (resp., also) at the “terminal optima/optimum” that will be detailed later in Appendix C-B while determined/computed at lines 11-30 of this algorithm.

By A)e) of Theorem 2, the optimum of equality-constrained QP is unique, iff ${\mathcal{N}}(A)\cap{\mathcal{N}}(P)=\{{\bf 0}\}$, or equivalently $V_{2}^{T}PV_{2}\succ 0$ in Eq. (13). Accordingly, further divide this case into whether $V_{2}^{T}PV_{2}$ is nonsingular.

i) If $V_{2}^{T}PV_{2}\succ 0$, then the unique optimum ${\bf x}^{\bar{*}}={\bf x}_{p}^{\bar{*}}$ is given by Eq. (13), where the freedom ${\boldsymbol{\varepsilon}}^{\bar{*}}\in{\mathcal{N}}(V_{2}^{T}PV_{2})=\{{\bf 0}\}$ is nulled, and included in line 3 of this algorithm. The associated optimal value $\bar{l}^{*}$ is given by Eq. (12), and included in line 4. However, only if this unique optimum resides in the feasible set ${\mathcal{F}}$, which is verified at line 5, the algorithm completes solving QP Problem (15), since this optimality $({\bf x}^{\bar{*}},\bar{l}^{*})$ under only equality constraints also solves the general QP Problem (15). It is worth mentioning that the pair of “$V_{2}^{T}PV_{2}\succ 0$ and the additional, returned parameter $\tilde{\mathcal{I}}^{*}=\emptyset$” serves as a unique identifier for this case, but this excludes the extreme case of singleton ${\mathcal{F}}\neq\emptyset$ (details at Remark 18 later).

ii) If $V_{2}^{T}PV_{2}$ is singular, then at first we need to check whether the optimal value is finite, whose equivalence condition is formulated in A)c) of Theorem 2 and included in line 7. If infinite, then the optimality of QP Problem (15) can only possibly occur at the “terminal optima/optimum”, owing to additional inequality constraints, which will be determined later in Appendix C-B. On the other hand, consider the finite optimal value as given by Eq. (12), with the associated optima by (13). These two have also been computed in lines 3 and 4 for a concise presentation. Note that, at this stage, only the particular solution of Eq. (13) is checked to be feasible or not; while the freedom arising from $V_{2}{\boldsymbol{\varepsilon}}^{\bar{*}}$, where ${\boldsymbol{\varepsilon}}^{\bar{*}}\in{\mathcal{N}}(V_{2}^{T}PV_{2})\neq\{{\bf 0}\}$, will be checked later in the for-all environment (at lines 12-29). This argument similarly follows ii)d) in Appendix C-B below. It is worth remarking that an expedited alternative is available by economizing the searching of any possible further optimum, as determined by some ${\boldsymbol{\varepsilon}}^{\bar{*}}\neq\{{\bf 0}\}$. Specifically, replacing lines 8-9 by 6.

C-B Cases (b)-(d) ${\bf x}_{p}^{\bar{*}}\not\in{\mathcal{F}}$

As illustrated in Figs. 3 (b)-(d), the remaining derivations correspond to lines 11-30. They examine all possible cases that the optimality occurs, corresponding to the boundary as determined by additional inequality constraint(s). It is worth noting that such a boundary can be either “edge” or “vertex”, depending on its dimension. Specifically, at line 12, we examine all the possible, ($2^{\kappa}-1$)’s subsets/cases ${\mathcal{I}}_{k}\subseteq{\mathcal{I}}$. The feasibility will be checked during the process, and each one is associated with the “augmented” equality-constrained QP problem with respect to $(\tilde{A},\tilde{\bf b})$ at line 13, where $\tilde{C}\in{}^{{\it k}\times{\it n}}$, $\tilde{A}\in{}^{({\it m}+{\it k})\times{\it n}}$, and $\tilde{\bf b}\in{}^{{\it m}+{\it k}}$.

In the very beginning, line 14 filters out the inconsistent case(s): augmented QP(s) imposed by inconsistent equality constraint(s), or not an “edge/vertex” of the polyhedron ${\mathcal{F}}$ as shown in Fig. 3. Obviously, the optimality does not occur in such case(s). The next stage, the if-else setting at lines 15-27, filters in only the cases of full-rank $\tilde{A}$, according to the formulation in (10); while the other(s) has/have already been considered before (this case/consideration), corresponding to the reduced, full-rank, equivalent counterpart. Also, the if-else setting distinguishes between augmented systems of many solutions (commonly an underdetermined system) and of the only one. Specifically, lines 15-20 mimic the processes at lines 1-8 but with respect to the “augmented” equality constraints $(\tilde{A},\tilde{\bf b})$, where $\tilde{V}_{2}\in{}^{{\it n}\times({\it n}-{\it m}-{\it k})}$ is an orthogonal matrix as computed at line 16; while lines 21-26 consider the full-column-rank $\tilde{A}$, and the unique feasible point/optimum candidate $\hat{\bf x}$ (computed at line 22), which is associated with some ${\mathcal{I}}_{j}$ and the only level set (corresponding to the optimal value for this augmented system). Note that:

1. i)

At lines 17 and 23, it is sufficient to check all those inequality constraints that are not “augmented/involved” in $(\tilde{A},\tilde{\bf b})$ corresponding to an ${\mathcal{I}}_{j}$. This is beneficial from a computational perspective, because excessive computations are avoided. 2. ii)

If the conditions at line 17 are all satisfied, then lines 18 and 19 gather the optimality candidate, as well as the involved/associated inequality constraints, in $\bar{\mathcal{L}}\times\bar{\mathcal{X}}\times\bar{\mathcal{I}}$. The last element provides the information on which “edge” the unique optimum (or the particular solution of optima) resides, with respect to the augmented equality-constrained QP. A geometric interpretation of this optimality candidate is illustrated in Fig. 4, with more details in Remark 16. On the other hand (when unsatisfied at line 17), according to Theorem 2, the consideration instead goes to one of the following:

1. a)

An unconstrained LP, as in B) of Theorem 2, the optimality can be determined in another case, say ${\mathcal{I}}_{\bar{j}}\supset{\mathcal{I}}_{j}$, if the optimality is actually associated with the ${\mathcal{I}}_{j}$ in this case. In other words, given the unbounded LP, the optimal value only happens by imposing further (inequality) constraints, and will be at the “terminal(s)”. Accordingly, the associated optimum/optima are further termed terminal. 2. b)

A constant problem, as in C) of Theorem 2. Obviously, its optimal value is just the only level set value. However, instead of checking whether the domain intersects ${\mathcal{F}}$, a more efficient way is to reserve this case to another one in the for-all environment, since there are only two possibilities: one is that this case will never be further constrained, which obviously does not solve QP Problem (15); regarding the other, the optimal value still remains and also occurs at the “terminal optimum/optima”, which is/are included at line 19 and later compared with all the other candidate(s) at line 30. This case is demonstrated by Example 10 (Sec. IV). 3. c)

An (equality-constrained) QP but unbounded below, as determined by A)c) of Theorem 2. This derivation is similar to ii)a) above and thus omitted. 4. d)

An (equality-constrained) QP of finite optimal value, with its unique optimum, or the particular solution of optima, outside of the polyhedron ${\mathcal{F}}$; in other words, not satisfying the other inequality constraint(s) that is/are not involved in this augmented system. Notably, there is still only one possibility that the optimality of QP Problem (15) is associated with this case, which corresponds to the nonuniqueness of optima. Specifically, this possibility is owing to the further imposed equality constraint(s), or the boundary of inequality constraint(s) from ${\mathcal{I}}\backslash{\mathcal{I}}_{j}$, on the degree of freedom of the optima with respect to the augmented QP under equality constraints and additional ones from ${\mathcal{I}}_{j}$. This will be elsewhere included/considered by virtue of the for-all environment in Algorithm 1. 3. iii)

Regarding the full-column-rank $\tilde{A}$, at lines 21-26, the only feasible point/optimum with respect to the augmented equality-constrained QP is computed a priori at line 22. If this point is also feasible with respect to the original/general QP Problem (15), as determined at line 23, then the corresponding, only, feasible/optimal value is later computed at line 24, and included as an optimality candidate at line 25 (the overall comparison in the end of this algorithm).

Finally, at line 30, the result concludes by comparing all the candidates in $\bar{\mathcal{L}}$ of finite elements. As a matter of fact, the cardinality of $\bar{\mathcal{L}}$ is at most $2^{\kappa}$, which is detailed later in Remark 17. The minimum/optimal value $\tilde{l}^{*}$ is associated with the corresponding terminal optimum/optima in $\bar{\mathcal{X}}$, which can be either the unique optimum, or the particular/terminal solution of nonunique optima, located on the “edge(s)” or “vertex/vertices” as determined by $\tilde{\mathcal{I}}^{*}$.

Remark 15

In the end of proof/algorithm, such an optimality pair $(\tilde{l}^{*},{\bf x}^{\tilde{*}},\tilde{\mathcal{I}}^{*})$ can be nonunique. A more in-depth interpretation is that the optimality can occur at the different/nonunique optima, or different “edges/vertices”. The information regarding the latter is completely gathered by means of this algorithm. As for the former, if the optimum is unique with respect to all possible $\tilde{\mathcal{I}}^{*}$(’s), then this information is also complete. On the other hand, if the optima (with respect to some $\tilde{\mathcal{I}}^{*}$) are nonunique, then, to remain focused of this presentation, preliminarily a subset is formulated using Algorithm 1. This subset includes all the terminal optima. That is, Algorithm 1 preliminarily gives closed-form solutions for all the optimum/optima at the terminals; while the intermediate points require specific attention due to the flexibility of inequality constraints. Although this indicates an interesting further research direction, Example 10 in Sec. IV demonstrates a potential of the proposed results for such a comprehensive generalization. That is, a closed-form solution to the considered QP, including an explicit representation for all the (nonunique) optima.

Remark 16

Fig. 4 demonstrates a fundamental concept “optimality candidates” for the design of Algorithm 1, in terms of the case where all ${\bf x}^{\bar{*}}\not\in{\mathcal{F}}$. It is worth noting that, according to Eq. (13), if there exists an ${\boldsymbol{\varepsilon}}^{\bar{*}}$ such that the corresponding ${\bf x}^{\bar{*}}\in{\mathcal{F}}$, then $\tilde{l}^{*}=\bar{l}^{*}$, and follow similar discussions in the remaining of this remark. Regarding this planar example of Fig. 4, the optimality occurs at (d) ${\bf x}^{\tilde{*}}$, and there are three representative cases/points (a)-(c) that are considered in the for-all design at lines 12-29 of Algorithm 1:

1. a)

The corresponding augmented pair $(\tilde{A},\tilde{\bf b})$ satisfies all the conditions at lines 15 and 17, except that its unique optimum is outside of the feasible set ${\mathcal{F}}$. Thus, this point is not an optimality candidate. 2. b)

The pair $(\tilde{A},\tilde{\bf b})$ satisfies all the conditions at lines 15 and 17, where its unique optimum is located on an edge of ${\mathcal{F}}$. Hence, this optimality candidate $\tilde{\bf x}_{p}^{\bar{*}}=\tilde{\bf x}^{\bar{*}}$ is included at line 19, for the overall comparison later at line 30. 3. c)

Since $\tilde{A}$ is of full column rank and its associated unique feasible point (optimum) $\hat{\bf x}$ is also feasible with regard to the other inequality constraints, $\hat{\bf x}$ is a candidate and also a vertex of ${\mathcal{F}}$, which is included at line 25 for the overall comparison (line 30).

However, the optimality of this demonstration occurs at the vertex/candidate (d) ${\bf x}^{\tilde{*}}$, whose arguments similarly follow the above (c). After comparing the associated, finitely many optimal values among all the candidates (line 30), including the ones at (b)-(d), the corresponding unique optimum ${\bf x}^{\tilde{*}}$, optimal (level set) value $\tilde{l}^{*}$, and information for the location of ${\bf x}^{\tilde{*}}$ ($\tilde{\mathcal{I}}^{*}$) are exactly determined. Note that the complete and explicit formulation of the QP example in Fig. 4 is not required for the discussions in this remark, particularly regarding the inequality constraints, since it is compactly two-dimensional for ease of reading.

Remark 17

This is to determine the largest possible cardinality of $\bar{\mathcal{L}}$. To start with, note that there are $2^{\kappa}$’s different subsets of ${\mathcal{I}}$. The case of empty set is considered in lines 3-10 to check the feasibility of ${\bf x}_{p}^{\bar{*}}$, which is associated with $\bar{\mathcal{L}}=\emptyset$ (as in the designed initial condition); while its possible freedom $(V_{2}{\boldsymbol{\varepsilon}}^{\bar{*}})$ is designed to be checked within the for-all environment (lines 12-29), if the solutions to QP Problems (10) and (15) intersect, such as the demonstrations (a) and (b) in Fig. 3. On the other hand, the for-all design/coverage includes all the other subsets (note that $k\geq 1$ at line 12), which is associated with, at most, $(2^{\kappa}-1)$’s elements for $\bar{\mathcal{L}}$.

Remark 18

An extreme example considers the singleton ${\mathcal{F}}\neq\emptyset$ (noting that the subset $\emptyset$ contains no point), and this remark explains how Algorithm 1 solves it. Similarly, divide the derivations into whether $\{{\bf x}_{p}^{\bar{*}}\}={\mathcal{F}}$. Regarding the equivalent/simpler case, then the solving process can be more efficiently completed without excessively going into the for-all environment (lines 12-29), while returning and ensuring the correct optimality. On the other hand, the process at lines 15-20 yields the optimality, only if $\tilde{\bf x}_{p}^{\bar{*}}\in{\mathcal{F}}$ for some ${\mathcal{I}}_{j}$; while that at 21-26 “exhaustively” includes this case, a vertex of ${\mathcal{F}}$. Although in the former (two) cases, this algorithm only concentrates on the particular solution ${\bf x}_{p}^{\bar{*}}/\tilde{\bf x}_{p}^{\bar{*}}$; nonetheless, this example also demonstrates that the consideration on the freedom $(V_{2}{\boldsymbol{\varepsilon}}^{\bar{*}}/\tilde{V}_{2}\tilde{\boldsymbol{\varepsilon}}^{\bar{*}})$ can be more than necessary, where, for brevity, the notation $\tilde{\boldsymbol{\varepsilon}}^{\bar{*}}$ similarly follows ${\boldsymbol{\varepsilon}}^{\bar{*}}$ but with respect to the augmented pair $(\tilde{A},\tilde{\bf b})$. As a comparison, it is not straightforward to apply most of the existing QP solvers [2, 20] to this singleton example, which are commonly/classically designed from the differential perspective while initiated by (knowing) a feasible point, such as the popular ASM [16].

Appendix D Proof of Theorem 3 (Extended QP)

A) By i) of Theorem 1, rewrite equivalently the CCQF at any level set value of $\check{l}$ as the following constrained convex quadratic equation (CCQE):

[TABLE]

According to Lemma 1, a necessary condition that renders the above CCQE in (40) solvable is

[TABLE]

Note that the constraint “${\bf q}\in{\mathcal{R}}(P)$” has been (implicitly) taken into consideration. The remaining part of this proof starts with the derivations for the sufficiency of Condition (41b). Accordingly, divide the derivations into whether the Hessian matrix is of full rank or not, as A-1) and A-2) below.

A-1) rank$(P)={\it n}$. In this case, we have i) $P$ is nonsingular, ii) ${\mathcal{R}}(P)={}^{\it n}$, and thus iii) the constraint “both ${\bf x},~{}{\bf q}\in{\mathcal{R}}(P)$” is automatically lifted, which ensures the sufficiency of Condition (41a). By virtue of Eq./Parameterization (4) under Condition (41a), the result thus follows.

A-2) rank$(P)={\it r}<{\it n}$. Let the SVD of $P$ be given by

[TABLE]

where $\tilde{U}_{1}\in{}^{{\it n}\times{\it r}}$, $\tilde{U}_{2}\in{}^{{\it n}\times({\it n-r})}$, and $\tilde{\Sigma}_{1}\in{}^{{\it r}\times{\it r}}$. For conciseness, the properties of this SVD (47) for derivations afterward are similar to, and thus referenced at, [14, Eqs. (35)-(37)]. Likewise, define $P^{\dagger}\coloneqq\tilde{U}_{1}\tilde{\Sigma}_{1}^{-1}\tilde{U}_{1}^{T}$ and $P^{\dagger/2}\coloneqq\tilde{U}_{1}\tilde{\Sigma}_{1}^{-1/2}\tilde{U}_{1}^{T}$. As a result, the CCQE in (40) is equivalent to the following CQE:

[TABLE]

where $\tilde{U}_{1}\tilde{U}_{1}^{T}+\tilde{U}_{2}\tilde{U}_{2}^{T}=I_{\it n}$ is inserted purposely, and $\check{\bf x}=\tilde{U}_{1}^{T}{\bf x}\in{}^{\it r}$ (resp., $\hat{\bf q}=\tilde{U}_{1}^{T}{\bf q}\in{}^{\it r}$) is the coordinate vector of ${\bf x}\in{\mathcal{R}}(P)$ (resp., ${\bf q}\in{\mathcal{R}}(P)$) with respect to the $\tilde{U}_{1}$-basis. Note that the original CCQF has been equivalently transformed into the CQE (48) with the full-rank Hessian matrix $\tilde{\Sigma}_{1}$, and the dimension is reduced from ${\it n}$ to ${\it r}$ (because of the constraint “both ${\bf x},~{}{\bf q}\in{\mathcal{R}}(P)$”). In addition, the solvability of the CQE (48) has already been ensured. To see this, by A) of Lemma 1, the CQE (48) is solvable, iff

[TABLE]

This has already been given/guaranteed according to Condition/Eq. (41b), and also ensures its sufficiency that renders the above CCQE (40) solvable.

Therefore, by virtue of Parameterization/Eq. (4), the solution set to the CQE in (48), where $\hat{\Sigma}_{1}$ is of full rank (${\it r}$), is given as

[TABLE]

where $\hat{\bf v}\in{}^{\it r}$ and $\|\hat{\bf v}\|=1$. In terms of the original coordinate, finally we obtain all the constrained solutions (namely, elements):

[TABLE]

where $\tilde{U}_{1}^{T}\tilde{U}_{1}=I_{\it r}$ is inserted purposely, $\check{\boldsymbol{\rho}}=\tilde{U}_{1}\hat{\bf v}\in{\mathcal{R}}(P)$, and $\|\check{\boldsymbol{\rho}}\|=\|\tilde{U}_{1}\hat{\bf v}\|=\|\hat{\bf v}\|=1$. Remarkably, the constraint ${\bf x}\in{\mathcal{R}}(P)$ is satisfied since the two vectors in Eq. (17), $P^{\dagger}{\bf q}$ and $P^{\dagger/2}\check{\boldsymbol{\rho}}$, reside in ${\mathcal{R}}(P)$.

B) The optimal value is finite, whose arguments similarly follow the proof for ii) of Theorem 1, and is mainly due to the constraint “${\bf q}\in{\mathcal{R}}(P)$”. Moreover, according to the above derivations for A), the necessary condition in Eq. (41b) is also sufficient to render the CCQE (40) solvable. Thus, the optimal value $\check{l}^{*}$ readily follows from the Condition (41b). Substituting $\check{l}^{*}$ into Eq. (17) yields the associated unique optimum ${\bf x}^{\hat{*}}$, which completes the proof.

Remark 19

A geometric interpretation of the preimage, as addressed in A) of Theorem 3, can be similarly inferred from Fig. 2, where the solution freedom residing in ${\mathcal{N}}(P)$ is not available/effective. This is essentially due to the constraint “${\bf x}\in{\mathcal{R}}(P)$” in Problem (16), while the other “${\bf q}\in{\mathcal{R}}(P)$” mainly renders this optimization problem with a finite optimal value. From a different viewpoint, this remark is also consistent with Eq. (7), whose derivations are similar and hence omitted for conciseness.

Appendix E Tables for all ${\mathcal{I}}_{j}$’s in Examples 9 and 10 (Sec. IV)

This appendix is designed to facilitate comparisons between Examples 9 and 10. Table II (resp., III) is associated with Example 9 (resp., 10), which demonstrates the solving process on a positive definite Hessian matrix given in [16] (resp., positive semidefinite and singular Hessian matrix), to summarize the details/steps when examining each case ${\mathcal{I}}_{j}$ according to Algorithm 1, specifically, the for-all environment at lines 12-29. Note that the case of subset/empty set “$\emptyset$” (in other words, no inequality constraint imposed) has been more efficiently considered a priori, in the beginning of the two examples (lines 2-10 in Algorithm 1). Moreover, ${\mathcal{I}}_{7}=\{1,2,3\}$ imposed by all the three inequality constraints is excluded in both tables/examples, for compactness, because it is obviously not associated with the optimality ($\tilde{\bf b}\not\in{\mathcal{R}}(\tilde{A})$).

Acknowledgment

The authors acknowledge Julian Mellor and Scott Read for their assistance in the native use of this language. This work was supported in part by the Max Planck Society, Germany; Ministry of Science and Technology, Taiwan; and Delta Electronics, Inc.