Robust Quadratic Programming with Mixed-Integer Uncertainty
Areesh Mittal, Can Gokalp, Grani A. Hanasusanto
TL;DR
This paper develops copositive programming reformulations and SDP approximations for robust quadratic programs with mixed-integer uncertainty, providing more effective solutions than existing methods.
Contribution
It introduces exact copositive reformulations and a conservative SDP approximation for mixed-integer robust quadratic programs, extending previous continuous-only approaches.
Findings
SDP approximation outperforms S-lemma-based methods
Reformulations applicable to two-stage problems with recourse
Demonstrated effectiveness on practical optimization problems
Abstract
We study robust convex quadratic programs where the uncertain problem parameters can contain both continuous and integer components. Under the natural boundedness assumption on the uncertainty set, we show that the generic problems are amenable to exact copositive programming reformulations of polynomial size. These convex optimization problems are NP-hard but admit a conservative semidefinite programming (SDP) approximation that can be solved efficiently. We prove that the popular approximate S-lemma method --- which is valid only in the case of continuous uncertainty --- is weaker than our approximation. We also show that all results can be extended to the two-stage robust quadratic optimization setting if the problem has complete recourse. We assess the effectiveness of our proposed SDP reformulations and demonstrate their superiority over the state-of-the-art solution schemes on…
| Objective gap | |||
|---|---|---|---|
| Statistic | SDP | -lemma | B&S |
| Mean | 0.0% | 108.4% | 99.7% |
| 10th Percentile | 0.0% | 93.9% | 80.5% |
| 90th Percentile | 0.0% | 119.6% | 115.3% |
| Benders | SDP | -lemma | B&S | |
|---|---|---|---|---|
| Mean solution time (in secs) | 626.9 | 10.2 | 0.45 | 0.004 |
| Objective gap | Suboptimality | |||
|---|---|---|---|---|
| Statistic | SDP | LDR | SDP | LDR |
| Mean | 2.7% | 26.9% | 1.7% | 10.0% |
| 10th Percentile | 2.0% | 23.8% | 1.3% | 7.1% |
| 90th Percentile | 3.2% | 30.2% | 2.2% | 12.8% |
| Benders | SDP | LDR | |
| Mean solution time (in secs) | 518.0 | 85.0 | 0.16 |
| Objective gap | Suboptimality | |||
|---|---|---|---|---|
| Statistic | SDP True | SDP Cont | SDP True | SDP Cont |
| Mean | 13.1% | 85.2% | 13.0% | 84.9% |
| 10th Percentile | 0.0% | 25.8% | 0.0% | 25.7% |
| 90th Percentile | 28.4% | 173.7% | 27.6% | 173.5% |
| Benders | SDP True | SDP Cont | |
| Mean solution time (in secs) | 52.9 | 11.3 | 0.63 |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
\newfloatcommand
capbtabboxtable[][\FBwidth]
Robust Quadratic Programming with Mixed-Integer Uncertainty
Areesh Mittal
Can Gokalp
Grani A. Hanasusanto
Abstract
We study robust convex quadratic programs where the uncertain problem parameters can contain both continuous and integer components. Under the natural boundedness assumption on the uncertainty set, we show that the generic problems are amenable to exact copositive programming reformulations of polynomial size. These convex optimization problems are NP-hard but admit a conservative semidefinite programming (SDP) approximation that can be solved efficiently. We prove that the popular approximate -lemma method—which is valid only in the case of continuous uncertainty—is weaker than our approximation. We also show that all results can be extended to the two-stage robust quadratic optimization setting if the problem has complete recourse. We assess the effectiveness of our proposed SDP reformulations and demonstrate their superiority over the state-of-the-art solution schemes on instances of least squares, project management, and multi-item newsvendor problems.
1 Introduction
A wide variety of decision making problems in engineering, physical, or economic systems can be formulated as convex quadratic programs of the form
[TABLE]
Here, is the feasible set of the decision vector and is assumed to be described by a polytope, is a vector of exogenous problem parameters, and are matrix- and vector-valued affine functions, respectively, while is a convex quadratic function. The objective of problem (1) is to determine the best decision that minimizes the quadratic function . The generic formulation (1) includes the class of linear programming problems [42] as a special case (when ), and has numerous important applications, e.g., in portfolio optimization [37], least squares regression [26], supervised classification [15], optimal control [41], etc. In addition to their exceptional modeling power, quadratic optimization problems of the form (1) are attractive as they can be solved efficiently using standard off-the-shelf solvers.
In many situations of practical interest, the exact values of the parameters are unknown when the decisions are made and can only be estimated through limited historical data. Thus, they are subject to potentially significant errors that can adversely impact the out-of-sample performance of an optimal solution . One popular approach to address decision problems under uncertainty is via robust optimization [2]. In this setting, we assume that the vector of uncertain parameters lies within a prescribed uncertainty set and we replace the objective function of (1) with the worst-case function given by
[TABLE]
This optimization problem yields a solution that minimizes the quadratic objective function under the most adverse uncertain parameter realization .
Robust optimization models are appealing as they require minimal assumptions on the description of the uncertain parameters and because they often lead to efficient solution schemes. In a linear programming setting, the resulting robust optimization problems are tractable for many relevant uncertainty sets and have been broadly applied to problems in engineering, finance, machine learning, and operations management [4, 6, 27]. Tractable reformulations for robust quadratic programming problems are derived in [25, 36] for the particular case when the quadratic functions (in ) exhibit a concave dependency in the uncertain parameters . When the functions are convex in both and as we consider in this paper, the corresponding robust problems are generically NP-hard if the uncertainty set is defined by a polytope, but become tractable—by virtue of the exact -lemma—if the uncertainty set is defined by an ellipsoid [4, 23]. Tractable approximation schemes have also been proposed for the standard setting that we consider in this paper. If the uncertainty set is described by a finite intersection of ellipsoids then a conservative semidefinite programming (SDP) reformulation is obtained by leveraging the approximate -lemma [5]. In [7], a special class of functions is introduced to approximate the quadratic terms in (2). The arising robust optimization problems are tractable if the uncertainty sets are defined through affinely transformed norm balls. In [36], conservative and progressive SDP approximations are devised by replacing each quadratic term in (2) with linear upper and lower bounds, respectively.
Most of the existing literature in robust optimization assume that the uncertain problem parameters are continuous and reside in a tractable conic representable set . However, certain applications require the use of mixed-integer uncertainty. Such decision problems arise prominently in the supply chain context where demands of non-perishable products are more naturally represented as integer quantities and in the discrete choice modeling context where the outcomes are chosen from a discrete set of alternatives. Other pertinent examples include robust optimization applications in logistic regression [43], classification problems with noisy labels [13, 51] and network optimization [1, 48]. If the uncertain parameters contain mixed-integer components then the problem becomes computationally formidable even in the simplest setting. Specifically, if all functions are affine in and the uncertain problem parameters are described by binary vectors, then computing the worst-case values in (2) is already NP-hard [21]. The corresponding robust version of (1) is tractable only in a few contrived situations, e.g., when the uncertainty set possesses a total unimodularity property or is described by the convex hull of polynomially many integer vectors [4]. Perhaps due to these limitations, there are currently very few results in the literature that provide a systematic and rigorous way to handle generic robust optimization problems with mixed-integer uncertainty. In this paper, we first reformulate the original problem as an equivalent finite-dimensional conic program of polynomial size, which absorbs all the difficulty in its cone, and then replace the cone with tractable inner approximations. An alternate way to handle integer uncertain parameters can be to solve the problem by simply ignoring the integrality assumption. However, doing so adds undesired conservativeness to the uncertainty set. Indeed, in our numerical experiments, we demonstrate that ignoring the integrality assumption on the uncertain parameters leads to overly conservative solutions.
Optimization problems under uncertainty may also involve adaptive recourse decisions which are taken once the uncertain parameters are realized [2, 46]. This setting gives rise to difficult min-max-min optimization problems which are generically NP-hard even if both the first- and the second-stage cost functions are affine in and [3]. Thus, they can only be solved approximately, either by employing discretization schemes which approximate the continuum of the uncertainty space with finitely many points [28, 31, 45] or by employing decision rule methods, which restrict the set of all possible recourse decisions to simpler parametric forms in [3, 22, 24]. We refer the reader to [17] for a comprehensive review of recent results in adaptive robust optimization. In this paper, we consider two-stage robust optimization problems with quadratic first- and second-stage objective function and a mixed-integer uncertainty set. We show that if the problem has complete recourse, then it can be reformulated as a conic program—which is amenable to tractable approximations.
The conic programming route that we take here to model optimization problems under uncertainty has previously been traversed. In [39], completely positive programming reformulations are derived to compute best-case expectations of mixed zero-one linear programs under first- and second-order moment information on the joint distributions of the uncertain parameters. This result has been extended and applied to other pertinent settings such as in stochastic appointment scheduling problems, discrete choice models, random walks and sequencing problems [32, 34, 38]. Recently, equivalent copositive programming reformulations are derived for generic two-stage robust linear programs [29, 50]. The resulting optimization problems are amenable to conservative semidefinite programming reformulations which are often stronger than the ones obtained from employing quadratic decision rules on the recourse function. In [20], the authors provide completely positive reformulation for a two-stage distributionally robust supply chain risk mitigation problem. They allow some components of to be binary, but assume precise knowledge of the first- and the second-order moments of the distribution of . The objective function that they consider is quadratic in the second-stage decision variables but affine in . In contrast to [20], we assume no information about the distribution of , other than the support. Furthermore, we allow the objective function to be quadratic in the decision variables, as well as in , which helps us model a more general class of robust problems, e.g., robust least squares [23].
In this paper, we advance the state-of-the-art in robust optimization along several directions. We summarize our main contributions as follows:
We prove that any robust convex quadratic program can be reformulated as a copositive program of polynomial size if the uncertainty set is given by a bounded mixed-integer polytope. We further show that the exactness result can be extended to the two-stage robust quadratic optimization setting if the problem has complete recourse. 2. 2.
By employing the hierarchies of semidefinite representable cones to approximate the copositive cones, we obtain sequences of tractable conservative approximations for the robust problem. These approximations can be made to have any arbitrary accuracy. We prove that even the simplest of these approximations is stronger than the well-known approximate -lemma method if the problem instance has only continuous uncertain parameters. Furthermore, when some uncertain parameters are restricted to take integer values, the approximate -lemma method is not applicable, while our method still generates a high-quality conservative solution. 3. 3.
We compare our approximation method to other state-of-the-art approximation schemes through extensive numerical experiments. We show that our approximation method generates better estimates of worst-case cost and yields less conservative solutions. We also demonstrate that ignoring the integrality assumption on the uncertainty set may lead to inferior solutions to the robust problem. 4. 4.
To the best of our knowledge, we are the first to provide an exact conic programming reformulation and to propose tractable semidefinite programming approximations for well-established classes of one-stage and two-stage robust quadratic programs.
The remainder of the paper is structured as follows. We formulate and discuss the generic robust quadratic programs in Section 2. We then derive the copositive programming reformulation in Section 3. Section 4 develops a conservative SDP reformulation and provides a theoretical comparison with the popular approximate -lemma method. In Section 5, we extend the results of Section 3 along several directions including two-stage robust quadratic optimization. We demonstrate the impact of our proposed reformulation via numerical experiments in Section 6, and finally, we conclude in Section 7.
Notation:
We use to denote the set of (non-negative) integers. For any positive integer , we use to denote the index set . We use to denote the -norm. We drop the subscript and write when referring to the -norm. The identity matrix and the vector of all ones are denoted by and , respectively. The dimension of such matrices will be clear from the context. We denote by the trace of a square matrix . For a vector , denotes the diagonal matrix with on its diagonal; whereas for a square matrix , denotes the vector comprising the diagonal elements of . We define as the Hadamard product (element-wise product) of two matrices and of the same size. For any integer , we define as the vector comprising all -th powers of 2, for . We define by () the space of all symmetric (positive semidefinite) matrices in . The cone of copositive matrices is denoted by , while its dual cone, the cone of completely positive matrices, is denoted by , where [44]. For any , the relations , , and indicate that is an element of , , and , respectively.
2 Problem Formulation
We study robust convex quadratic programs (RQPs) of the form
[TABLE]
where the set and the functions , , and have the same definitions as those in (1). The vector comprises all the uncertain problem parameters and is assumed to belong to the uncertainty set given by a bounded mixed-integer polyhedral set
[TABLE]
where and . We assume without loss of generality that the first elements of are integer, while the remaining are continuous. Since is bounded, we may further assume that there exists a scalar integer such that for every . Note that the quantity is bounded by a polynomial function in the bit length of the description of and .
Example 1** (Robust Portfolio Optimization).**
Consider the classical Markowitz mean-variance portfolio optimization problem
[TABLE]
where is the unit simplex in , is the prescribed risk tolerance level of the investor, while and are the true mean and covariance matrix of the asset returns, respectively. The objective of this problem is to determine the best vector of weights that maximizes the mean portfolio return and that also minimizes the portfolio risk that is captured by the variance term . Here, the trade-off between these two terms is controlled by the scalar in the objective function.
In practice, the true values of the parameters and are unknown and can only be estimated by using the available historical asset returns , as follows:
[TABLE]
In the robust optimization setting, we assume that the precise location of each sample point is uncertain and is only known to belong to a prescribed uncertainty set containing . To bring the resulting problem into the standard form (3), we introduce the expanded uncertainty set
[TABLE]
comprising the terms and , . Using this uncertainty set, we arrive at the following robust version of (5):
[TABLE]
This problem constitutes an instance of (3) with the input parameters
[TABLE]
Example 2** (Robust Project Crashing).**
Consider a project that is described by an activity-on-arc network , where is the set of nodes representing the events, while is the set of arcs representing the activities. We assume that that node with index represents the start of the project and the node with index represents the end of the project. We define to be the nominal duration of the activity . Here, we assume that the durations , , are already normalized so that they take values in the unit interval.
The goal of project crashing is to determine the best resource assignments , , on the activities that minimize the project completion time or makespan. If the activity duration represents the length of the arc , then the project completion time can be determined by computing the length of the longest path from the start node to the end node. We can formulate project crashing as the optimization problem
[TABLE]
where
[TABLE]
If the task durations are uncertain and are only known to belong to the prescribed uncertainty set , then we arrive at the robust optimization problem
[TABLE]
By combining the suprema over and , and linearizing the bilinear terms , , we can reformulate the objective of this problem as
[TABLE]
where
[TABLE]
Using the new objective function (7) and uncertainty set (8), the resulting robust optimization problem constitutes an instance of (3) with the input parameters , , and .
In the remainder of the paper, for any fixed , we define the mixed-integer quadratic program
[TABLE]
which corresponds to the inner subproblem in the objective of (3). We may therefore represent (3) as
[TABLE]
In the next section, we derive exact copositive programming reformulation for evaluating . By substituting with the emerging copositive program, we obtain an equivalent finite-dimensional convex reformulation for the RQP (3) that is principally amenable to numerical solution.
3 Copositive Programming Reformulation
In this section, we derive an equivalent copositive programming reformulation for (3) by adopting the following steps. For any fixed , we first derive a copositive upper bound on . We then show that the resulting reformulation is in fact exact under the boundedness assumption on the uncertainty set .
3.1 A Copositive Upper Bound on
To derive the copositive reformulation, we leverage the following result by Burer [11] which enables us to reduce a generic mixed-binary quadratic program into an equivalent conic program of polynomial size.
Theorem 1** ([11, Theorem 2.6]).**
The mixed-binary quadratic program
[TABLE]
is equivalent to the completely positive program
[TABLE]
where , and it is implicitly assumed that , for any satisfying .
We also rely on the following standard result which allows us to represent a scalar integer variable using only logarithmically many binary variables [47].
Lemma 1**.**
If is a scalar integer decision variable taking values in , with , then we can reformulate it concisely by employing binary decision variables , as follows:
[TABLE]
Using Theorem 1 and Lemma 1, we are now ready to state our first result.
Proposition 1**.**
For any fixed decision the optimal value of the quadratic maximization problem (9) coincides with the optimal value of the completely positive program
[TABLE]
where
[TABLE]
with
[TABLE]
Proof.
Lemma 1 enables us to reformulate the mixed-integer quadratic program (9) equivalently as the mixed-binary quadratic program
[TABLE]
We now employ Theorem 1 to derive the equivalent completely positive program for (13). To this end, we first bring the above quadratic program into the standard form (10). We introduce the redundant linear constraints , , which are pertinent for the exactness of the reformulation, and we define new auxiliary slack variables , , to transform these inequalities into the equality constraints , . This yields the equivalent problem
[TABLE]
We next define the expanded vector
[TABLE]
that comprises all decision variables in (14). Together with the augmented parameters (12), we can reformulate (14) concisely as the problem
[TABLE]
The mixed-binary quadratic program (15) already has the desired standard form (10) with inputs , , , , , and . We may thus apply Theorem 1 to obtain the equivalent completely positive program (11). This completes the proof. ∎
We remark that in view of the concise representation in Lemma 1, the size of the completely positive program (11) remains polynomial in the size of the input data. This completely positive program admits a dual copositive program given by
[TABLE]
By weak conic duality, the optimal value of this copositive program constitutes an upper bound on .
Proposition 2**.**
For any fixed decision we have .
3.2 A Copositive Reformulation of RQP
In this section, we demonstrate strong duality for the primal and dual pair (11) and (16), respectively, under the natural boundedness assumption on the uncertainty set . This exactness result enables us to reformulate the RQP (3) equivalently as a copositive program of polynomial size.
Theorem 2** (Strong Duality).**
For any fixed decision we have .
We would like to mention that a similar result is proved in a recent paper (Theorem 8 in [9]). However, the two proofs are quite different from one another. While the proof in [9] establishes strong duality by proving the existence of a Slater point for a general copositive program, we show explicitly how to construct a Slater point for the copositive program (16) from input parameters. Because of its constructive nature, we believe our proof is interesting on its own and sheds some light on the geometry of the feasible region of (16).
We note that the primal completely positive program (11) never has an interior [12]. In order to prove Theorem 2, we construct a Slater point for the dual copositive program (16). The construction of the Slater point for problem (16) relies on the following two lemmas. We observe that by construction the boundedness of the uncertainty set means that the lifted polytope
[TABLE]
is also bounded. This gives rise to the following lemma on the strict copositivity of the matrix .
Lemma 2**.**
We have .
Proof.
The boundedness assumption implies that the recession cone of the set coincides with the point , that is, . Thus, for every , , we must have , which further implies that for all such that . Hence, the matrix is strictly copositive. ∎
The next lemma, which was proven in [29, Lemma 4], constitutes an extension of the Schur complements lemma for matrices with a copositive sub-matrix. We include the proof here to keep the paper self-contained.
Lemma 3** (Copositive Schur Complements).**
Consider the symmetric matrix
[TABLE]
We then have if and .
Proof.
Consider a non-negative vector satisfying . We have
[TABLE]
The final inequality follows from the assumptions , and . In fact, the inequality will be strict, which can be shown by considering the following two cases:
If , then . Therefore , which implies that . 2. 2.
If , then the assumption implies that .
Therefore, in both cases, by rescaling we have for all such that . Hence, . ∎
Using Lemmas 2 and 3, we are now ready to prove the main strong duality result.
Proof of Theorem 2.
We construct a Slater point for problem (16). Specifically, we set , , and for some . Problem (16) then admits a Slater point if there exist scalars , such that
[TABLE]
Lemma 2 implies that for a sufficiently large the matrix is strictly copositive. Thus, we can choose a positive to ensure that
[TABLE]
Using Lemma 3, we may conclude that the strict copositivity constraint in (18) is satisfied by the constructed solution . Thus, problem (16) admits a Slater point and strong duality indeed holds for the primal and dual pair (11) and (16), respectively. ∎
The exactness result portrayed in Theorem 2 enables us to derive the equivalent copositive programming reformulation for (3).
Theorem 3**.**
The RQP (3) is equivalent to the following copositive program.
[TABLE]
The proof of Theorem 3 relies on the following lemma, which linearizes the quadratic term in the left-hand side matrix of problem (16).
Lemma 4**.**
Let be a symmetric matrix and be an arbitrary matrix with . Then the copositive inequality
[TABLE]
is satisfied if and only if there exists a positive semidefinite matrix such that
[TABLE]
Proof.
The only if statement is satisfied immediately by setting . To prove the converse statement, assume that there exists such a positive semidefinite matrix . Then by the Schur complement the semidefinite inequality in (21) implies that and, a fortiori, . Combining this with the copositive inequality in (21) then yields (20). Thus, the claim follows. ∎
Proof of Theorem 3.
Applying Theorem 2, we may replace the objective function of (3) with the corresponding copositive reformulation, we thus find that problem (3) is equivalent to
[TABLE]
Next, we apply Lemma 4 to linearize the quadratic terms , which gives rise to the desired copositive program (19). This completes the proof. ∎
4 Conservative Semidefinite Programming Approximation
The copositive program (19) is intractable due to its equivalence with generic RQPs over a polyhedral uncertainty set [4]. In the copositive reformulation, however, all the difficulty of the original problem (3) is shifted into the copositive cone , which has been well-studied in the literature. Specifically, there exists a hierarchy of increasingly tight semidefinite representable inner approximations that converge in finitely many iterations to [40, 10, 16, 33]. The simplest of these approximations is given by the cone
[TABLE]
which contains all symmetric matrices that can be decomposed into a sum of positive semidefinite and non-negative matrices. For dimensions it can be shown that [18], while for , is a strict subset of .
Replacing the cone in (19) with the inner approximation gives rise to a tractable conservative approximation for the RQP (3). In this case, however, the resulting optimization problem might have no interior or even become infeasible as the Slater point constructed in Theorem 2 can fail to be a Slater point to the restricted problem. Indeed, the strict copositivity of the matrix is in general insufficient to ensure that the matrix is also strictly positive definite. To remedy this shortcoming, we suggest the following simple modification to the primal completely positive formulation of in (11). Specifically, we assume that there exists a non-degenerate ellipsoid centered at with radius and shape parameter given by
[TABLE]
that contains the lifted set in (17). We then consider the following augmented completely positive programming reformulation for the maximization problem (9).
[TABLE]
Here, we have added the redundant constraint to (11), which arises from linearizing the quadratic constraint
[TABLE]
where we have set . The dual of the augmented problem (22) is given by the following copositive program.
[TABLE]
Note that we have since all the new additional terms are redundant for the original reformulations. Nevertheless, since the ellipsoid is non-degenerate, we find that the matrix is positive definite. We can thus set all eigenvalues of the scaled matrix to any arbitrarily large positive values by controlling the scalar . This suggests that replacing the cone with its inner approximation in (23) will always yield a problem with a Slater point.
Apart from helping us prove the existence of a Slater point, adding an ellipsoidal constraint to the description of the uncertainty set can also be of help numerically. Although, the constraint is redundant for the exact problem, it might not be redundant for the conservative approximation obtained by replacing with . Adding the constraint results in an additional variable in the SDP approximation, which can improve the objective value. Ideally, we would like the volume of the ellipsoid to be as small as possible to get more improvement. However, determining the parameters of the ellipsoid having minimum volume that encloses the set is NP-hard. A feasible ellipsoid that can be generated tractably is , where
[TABLE]
Note that the parameter of the ellipsoid can be determined by solving linear programs. Depending on the specific uncertainty set at hand, it might be possible to find other tighter ellipsoidal approximations.
4.1 Comparison with the Approximate -lemma Method
Next, we show that solving the problem by replacing with the simplest inner approximation is better than the approximate -lemma method. Since the latter is only valid in the case of continuous uncertain parameters, we restrict the discussion to the case where the bounded uncertainty set contains no integral terms and is given by the polytope . Here, the extended parameters (12) simplify to
[TABLE]
while the maximization problem (9) reduces to
[TABLE]
The copositive programming reformulation (23) can then be simplified to
[TABLE]
Replacing the cone in (25) with its inner approximation , we obtain a tractable SDP reformulation whose optimal value constitutes an upper bound on . Alternatively, we describe the approximate -lemma method below, which provides a different conservative SDP approximation for (24).
Proposition 3** (Approximate -lemma Method [4]).**
Assume that the uncertainty set is a bounded polytope and there is an ellipsoid centered at of radius given by that contains the set . Then, for any fixed , the maximization problem (9) is upper bounded by the optimal value of the following semidefinite program:
[TABLE]
Proof.
The quadratic maximization problem in (24) can be equivalently reformulated as
[TABLE]
Here, the last constraint is added without loss generality since . Reformulating the problem into its Lagrangian form then yields
[TABLE]
where the inequality follows from the weak Lagrangian duality. We next introduce an epigraphical variable that shifts the supremum in the objective function into the constraint. We have
[TABLE]
Reformulating the semi-infinite constraint as a semidefinite constraint then yields the desired reformulation (26). This completes the proof. ∎
The next proposition shows that the approximation resulting from replacing the copositive cone in (25) with its coarsest inner approximation is stronger than the state-of-art approximate -lemma method.
Proposition 4**.**
The following relation holds.
[TABLE]
Proof.
The equality and the first inequality hold by construction. To prove the second inequality, we consider the following semidefinite program that arises from replacing the cone with the inner approximation in (25).
[TABLE]
Next, we show that any feasible solution to (26) can be used to construct a feasible solution to (27) with the same objective value. Specifically, we set , , , , , , and . The feasibility of the solution in (26) then implies that the constructed solution is also feasible in (27). One can verify that these solutions give rise to the same objective function value for the respective problems. Thus, the claim follows. ∎
Next, we demonstrate that the inequality in in Proposition 4 can often be strict. This affirms that the proposed SDP approximation (27) is indeed stronger than the approximate -lemma method.
Example 3**.**
Consider the following quadratic maximization problem:
[TABLE]
A simple analysis shows that , which is attained at the solution . The problem (28) constitutes an instance of problem (24) with the parameterizations
[TABLE]
Here, the uncertainty set is given by the polytope , which corresponds to the inputs and . Replacing the cone with its inner approximation in the copositive programming reformulation of (28), we find that the resulting semidefinite program yields the same optimal objective value of . Meanwhile, the corresponding approximate -lemma method yields an optimal objective value . Thus, while the SDP approximation of the copositive program (25) is tight, the approximate -lemma generates an inferior objective value for the simple instance (28).
5 Extensions
In this section, we discuss several extensions to the RQP (3) which are also amenable to exact copositive programming reformulation. In Section 5.1, we study two-stage robust optimization with mixed-integer uncertainty set where the objective is quadratic in the first- and the second-stage decision variables. In Section 5.2, we develop an extension to the case when the model has robust quadratic constraints. Finally, in Section 5.3, we discuss the case where the objective function contains quadratic terms which are not convex in the uncertain parameter vector .
5.1 Two-Stage Robust Quadratic Optimization
In this section, we study the two-stage robust quadratic optimization problems of the form
[TABLE]
Here, for any fixed decision and uncertain parameter realization , the second-stage cost coincides with the optimal value of the convex quadratic program given by
[TABLE]
where and are matrix- and vector-valued affine functions, respectively.
Example 4** (Support Vector Machines with Noisy Labels).**
Consider the following soft-margin support vector machines (SVM) model for data classification.
[TABLE]
Here, for every index , the vector is a data point that has been labeled as . The objective of problem (31) is to find a hyperplane that separates all points labeled with the ones labeled . If the hyperplane satisfies , , then the data points are linearly separable. In practice, however, these data points may not be linearly separable. We thus seek the best linear separator that minimizes the number of incorrect classifications. This non-convex objective is captured by employing the hinge loss term in (31) as a convex surrogate. Here, the term in the objective function constitutes a regularizer for the coefficient .
If the labels are erroneous, then one could envisage a robust optimization model that seeks the best linear separator in view of the most adverse realization of the labels. To this end, we assume that the vector of labels is only known to reside in a prescribed binary uncertainty set . Then an SVM model that is robust against uncertainty in the labels can be formulated as
[TABLE]
where
[TABLE]
This problem constitutes an instance of (29) with the decision vector , and the input parameters
[TABLE]
The exactness result portrayed in Theorems 2 and 3 can be extended to the two-stage robust optimization problem (29). Specifically, if the problem has a complete recourse111The two-stage problem (29) has complete recourse if there exists with , which implies that the second-stage subproblem is feasible for every and . then, by employing Theorem 2 and extending the techniques developed in [29, Theorem 4], the two-stage problem (29) can be reformulated as a copositive program of polynomial size.
Theorem 4**.**
Assume that has full column rank. Then the two-stage robust optimization problem (29) is equivalent to the copositive program
[TABLE]
where
[TABLE]
[TABLE]
with
[TABLE]
Proof.
Since has full column rank, the matrix is positive definite. Thus, for any fixed and , the recourse problem (30) admits a dual quadratic program given by
[TABLE]
Strong duality holds as the two-stage problem (29) has complete recourse. Substituting the dual formulation (33) into the objective of (29) yields
[TABLE]
Thus, for any fixed , the objective value of the two-stage problem (29) coincides with the optimal value of a quadratic maximization problem, which is amenable to an exact completely positive programming reformulation similar to the one derived in Proposition 1. We can then follow the same steps taken in the proofs of Theorems 2 and 3 to obtain the equivalent copositive program (32). This completes the proof. ∎
Remark 1**.**
The assumption that has full column rank in Theorem 4 can be relaxed. If does not have full column rank then the symmetric matrix is not positive definite but admits the eigendecomposition , where is an orthogonal matrix whose columns are the eigenvectors of , while is a diagonal matrix with the eigenvalues of on its main diagonal. We assume without loss of generality that the matrix has the block diagonal form
[TABLE]
where is a diagonal matrix whose main diagonal comprises the non-zero eigenvalues of . Next, by using the constructed eigendecomposition and performing the change of variable , we can reformulate the recourse problem (30) equivalently as
[TABLE]
where and . The dual of this problem is given by the following quadratic program with a linear constraint system:
[TABLE]
We can then repeat the same steps in the proof of Theorem 4 to obtain an equivalent copositive programming reformulation. We omit this result for the sake of brevity.
5.2 Robust Quadratically Constrained Quadratic Programming (RQCQP)
The setting that we consider can be extended to the case where, in addition to the robust quadratic objective function, there are several robust quadratic constraints of the form
[TABLE]
In this case, the goal is to find a decision which minimizes the worst-case objective function, while ensuring that the quadratic constraints are satisfied for all possible uncertain parameter vectors in .
For every , we define and similarly to the definitions of the extended parameters and in (12). By applying Theorem 2, the quadratic maximization problem in the -th constraint of (34) can be replaced with a copositive minimization problem, which yields the constraint
[TABLE]
The constraint is satisfied if and only if there exist decision variables , , , and such that the constraint system
[TABLE]
is satisfied. Therefore the -th constraint of (34) can be replaced by the constraint system (35). The procedure for linearization of the quadratic terms is analogous to the method presented in Theorem 3.
5.3 Non-Convex Terms in the Objective Function
All exactness results in this paper extend immediately to the setting where the objective function in (3) involves non-convex quadratic terms in the uncertainty . Specifically, we consider the objective function
[TABLE]
where is a matrix-valued affine function of . We can still use Theorem 1 to reformulate as the optimal value of a copositive program. By following the steps of Proposition 1 and Theorem 3, the copositive programming reformulation is obtained by replacing the last constraint in (19) with the copositive constraint
[TABLE]
where
[TABLE]
We omit the details for the sake of brevity.
6 Numerical Experiments
In this section, we assess the performance of the SDP approximations presented in Section 4. All optimization problems are solved using the YALMIP interface [35] on a 16-core 3.4 GHz computer with 32 GB RAM. We use MOSEK 8.1 to solve SDP formulations, and CPLEX 12.8 to solve integer programs and non-convex quadratic programs.
6.1 Least Squares
The classical least squares problem seeks an approximate solution to an overdetermined linear system which minimizes the residual . This yields the following quadratic program:
[TABLE]
The solution to this problem can be very sensitive to perturbations in the input data and [19, 26]. To address the issue of parameter uncertainty, El Ghaoui and Lebret [23] recommend solving the following robust optimization problem:
[TABLE]
Here, the goal is to find a solution that minimizes the worst-case residual when the matrix and the vector can vary within the prescribed uncertainty set . A tractable SDP reformulation of this problem is derived in [23] for problem instances where the uncertainty set is given by the Frobenius norm ball
[TABLE]
We consider the case when the uncertainty set is a polytope, and compare our SDP scheme with the state-of-the-art approximate -lemma method described in Section 4.1. We also compare our method with the approximation scheme proposed by Bertsimas and Sim [7], where the worst-case quadratic term in (2) is replaced with an upper bounding function. Minimizing this upper-bounding function over yields an approximate solution to the RQP. We note that the robust least squares problem can be solved to optimality using Benders’ constraint generation method [8]. However, doing so entails solving a non-convex quadratic optimization problem at each step to generate a valid cut, which becomes intractable when and become large.
In our experiment, we consider the case where the uncertainty affects only the right-hand side vector (i.e., ). We assume that the uncertain parameter depends affinely on factors represented by , where . Specifically, we consider the uncertainty set
[TABLE]
where is the factor matrix and lies in the interval . By substituting and into (36), the resulting robust problem constitutes an instance of RQP (3) with the following input parameters:
[TABLE]
In order to solve the problem using our method, we modify the formulation discussed in Section 4 slightly, which leads to a tremendous reduction in the solution time. We discuss this modification in Appendix A.
We perform an experiment on problem instances of dimensions , and . The experimental results are averaged over random trials generated in the following manner. In each trial, we sample the matrix and the vector from the uniform distribution on and , respectively. Each row of the matrix is sampled randomly from a standard simplex, and is generated uniformly at random from the interval . For problems of this size, we are unable to solve the problem to optimality using Benders’ method as the solver runs out of memory. Therefore, we put a time limit of seconds for each iteration of the Benders’ method. By doing so, Benders’ method yields a lower bound to the optimal worst-case residual, which we use as a baseline to compute the objective gaps for the approximation methods.
Table 1 summarizes the optimality gaps of the approximation methods. The results show that our method significantly outperforms the other two approximations in terms of the estimates of the worst-case residuals. While the other two approximations generate overly pessimistic estimates of the resulting worst-case residuals (with a relative difference of about 100%), the worst-case residuals estimated using our method have negligible objective gaps.
Table 2 reports the solution times of finding the exact solution (using Benders’ method) and the upper bounds provided by various approximation methods. It can be observed that the improvement in solution quality given by our method comes at the cost of longer solution times compared to other approximation methods. However, our method is still significantly faster than the exact Benders’ method. We also note that while the approximation scheme described in [7] can be solved quickly, it is only valid when the uncertainty set is defined as a norm-bounded set ( norm in our experiment). Our method, on the other hand, is applicable for general polyhedral uncertainty sets.
6.2 Project Management
In this experiment, we consider the project crashing problem described in Example 2, where the duration of activity is given by the uncertain quantity . Here, is the nominal activity duration and represents exogenous fluctuations. We consider randomly generated project networks of size and order strength 0.75,222The order strength denotes the fraction of all possible precedences between the nodes that are enforced in the graph (either directly or through transitivity). which gives rise to projects with an average of activities. Let be the amount of resources that are used to expedite the activity . We fix the feasible set of the resource allocation vector to , so that at most of the activities can receive the maximum resource allocation. The uncertainty set of is defined through a factor model as follows:
[TABLE]
where the factor size is fixed to . We set the nominal task durations to . In each trial, we sample the factor loading vector from the uniform distribution on , which ensures that the duration of each activity can deviate by up to of its nominal value. We can form the final mixed-integer uncertainty set from using the procedure described in Example 2 (Equation (8)).
In our experiment, we compare the performance of our proposed SDP approximation with linear decision rules (LDR) approximation scheme discussed in [14, 48] which we describe below. In Example 2, for our reformulation, we model the second stage problem as the maximization problem over the binary variables (See Equation (6)). Alternatively, the second-stage problem can be written as the following minimization problem:
[TABLE]
Here, is second-stage variable which depends on the realization of the uncertain . In the LDR approximation scheme, is restricted to be an affine function of , which yields a tractable conservative approximation. To assess the suboptimality of our SDP and the LDR approximation scheme, we solve the problem to optimality using Benders’ constraint generation method.
Table 3 presents the optimality gaps of the two approximation methods for randomly generated project networks. The solution times of all the methods are reported in Table 4. It can be observed that our proposed SDP approximation consistently provides near-optimal estimates of the worst-case project makespan ( gaps). On the other hand, while the LDR bound can be computed quickly, the bounds are too pessimistic ( gaps). The th and th percentiles of the objective gaps further indicate that the estimated makespan generated from our SDP approximation stochastically dominates the makespan generated from the LDR approximation. In addition to a higher estimate of the worst-case makespan, the actual makespan of the resource allocation generated by the LDR approximation is also higher than the ones generated by our method, as shown in the “Suboptimality” column in Table 3. The experimental results demonstrate that our method generates near-optimal solutions to the project crashing problem faster than solving the problem to optimality using Benders’ method.
6.3 Multi-Item Newsvendor
We now demonstrate the advantage of using a mixed-integer uncertainty set over using a continuous uncertainty set in a variant of the multi-item newsvendor problem, where an inventory planner must determine the vector of order quantities for different raw-materials at the beginning of a planning period. The raw materials are used to make different types of products which are then sold to customers. The matrix is such that represents the amount of raw material required to make unit of product . The demands for these products are uncertain and are assumed to belong to a prescribed discrete uncertainty set . We assume that there are no ordering costs on the raw materials but the total order quantity must not exceed a given budget . Excess inventory of the -th raw material incurs a per-unit holding cost of , while the unmet demand incurs a quadratic penalty with coefficient . The quadratic penalty on the unmet demand is added to discourage stock-outs [30, 49].
For any realization of the demand vector , the total cost of a fixed order is given by
[TABLE]
Here, we use the notation to denote . The objective of a risk-averse inventory planner is then to determine a vector of order quantities that minimizes the worst-case total cost . This gives rise to the optimization problem
[TABLE]
This problem constitutes an instance of the two-stage robust quadratic optimization problem (29) with parameters
[TABLE]
In this experiment, we compare the performance of the SDP approximation of the optimization problem (37) when is explicitly modeled as a discrete vector versus the model where the integer restriction on is ignored. We consider problems with raw materials and products. We fix the vector of holding costs to , the ordering budget to , and the penalty constant to . All experimental results are averaged over random trials generated in the following manner. We assume that every product uses one unit each of two randomly chosen raw materials. In each trial, we generate every element of uniformly at random from the interval . We define the actual discrete uncertainty set () and the set formed by ignoring the integrality assumption () as:
[TABLE]
and solve the SDP approximations of (37) with inputs and . We use the Benders’ constraint generation method to solve the problem to optimality.
The statistics of the optimality gaps generated by the models using and are reported in Table 5. The solution times of all the methods are presented in Table 6. We observe that the model using as the uncertainty set provides much better estimates of the worst-case cost ( average gap) than the model using ( average gap). Furthermore, our proposed SDP approximation can be solved much faster than solving the problem exactly using Benders’ method. For problems with integer uncertainty, these experimental results suggest that the SDP approximation which utilizes the integer restriction gives high-quality solutions in comparison to the approximation which neglects these restrictions.
7 Conclusion
The paper aims at developing a near-optimal approximation method for one- and two-stage robust quadratic programs with mixed-integer uncertain parameters. The approximation method developed in the paper is not only more general than the current state-of-the-art approximate -lemma method—since the latter only handles continuous uncertain parameters—but is guaranteed to yield a better estimate of the optimal value. Furthermore, our numerical experiments show that the difference in the performance of the two approximation method can be quite significant. Our experimental results also demonstrate the disadvantage of ignoring the integer restrictions on the uncertain parameters. In the future, it would be interesting to extend the model to the distributionally robust setting, where additional information about the distribution of the uncertain parameters is explicitly incorporated.
Appendix A Implementation of Least Squares in Section 6.1
In this section, we limit the discussion to the case when there are no discrete uncertain parameters. In the paper, we consider the uncertainty set to be of the standard form . However, in some cases, the uncertainty sets are more naturally represented in the inequality form . Transforming the uncertainty set in standard form involves introducing additional variables and constraints which increases the problem size. As an example, in the least squares experiment in Section 6.1, we consider the uncertainty set . By lifting, the uncertainty set can be equivalently written as , which is of the form . The paper [12] presents a generalized copositive programming (GCP) reformulation of non-convex quadratic programs over conic representable sets. In [50], the authors consider a conservative approximation when the cone is polyhedral, which is relevant for the polyhedral uncertainty sets that we consider. Utilizing this GCP-based approximation, the robust least squares problem
[TABLE]
that we consider in Section 6.1 yields the following conservative SDP approximation:
[TABLE]
We use this formulation with the uncertainty set for our experiment in Section 6.1. Skipping the conversion to the standard form generates same the objective value, but reduces the average solution time from seconds to about seconds. We emphasize that this alternate formulation might not be valid when some of the components of are restricted to be integers. Therefore, it is not straightforward to apply it to the project management and the newsvendor experiments, both of which contain discrete uncertain parameters.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] S. D. Ahipasaoglu, K. Natarajan, and D. Shi. Distributionally robust project crashing with partial or no correlation information. 2016.
- 2[2] A. Ben-Tal, L. El Ghaoui, and A. Nemirovski. Robust Optimization . Princeton University Press, 2009.
- 3[3] A. Ben-Tal, A. Goryashko, E. Guslitzer, and A. Nemirovski. Adjustable robust solutions of uncertain linear programs. Mathematical Programming A , 99(2):351–376, 2004.
- 4[4] A. Ben-Tal and A. Nemirovski. Robust convex optimization. Mathematics of Operations Research , 23(4):769–805, 1998.
- 5[5] A. Ben-Tal, A. Nemirovski, and C. Roos. Robust solutions of uncertain quadratic and conic-quadratic problems. SIAM Journal on Optimization , 13(2):535–560, 2002.
- 6[6] D. Bertsimas, D. B. Brown, and C. Caramanis. Theory and applications of robust optimization. SIAM review , 53(3):464–501, 2011.
- 7[7] D. Bertsimas and M. Sim. Tractable approximations to robust conic optimization problems. Mathematical Programming B , 107(1-2):5–36, 2006.
- 8[8] J.W. Blankenship and J. E. Falk. Infinitely constrained optimization problems. Journal of Optimization Theory and Applications , 19(2):261–281, 1976.