Exact Augmented Lagrangian Duality for Mixed Integer Quadratic Programming

TL;DR

This paper establishes theoretical guarantees for the augmented Lagrangian dual approach in solving mixed integer quadratic programming problems, showing conditions under which it achieves zero duality gap.

Contribution

It proves that the augmented Lagrangian dual can reach zero duality gap asymptotically and finitely for MIQP under certain conditions, with polynomial bounds on penalty weights.

Findings

ALD reaches zero duality gap asymptotically as penalty weight increases.

Finite penalty weights suffice for zero duality gap with any norm.

Polynomial bounds on penalty weights ensure zero duality gap.

Abstract

Mixed integer quadratic programming (MIQP) is the problem of minimizing a convex quadratic function over mixed integer points in a rational polyhedron. This paper focuses on the augmented Lagrangian dual (ALD) for MIQP. ALD augments the usual Lagrangian dual with a weighted nonlinear penalty on the dualized constraints. We first prove that ALD will reach a zero duality gap asymptotically as the weight on the penalty goes to infinity under some mild conditions on the penalty function. We next show that a finite penalty weight is enough for a zero gap when we use any norm as the penalty function. Finally, we prove a polynomially bound on the weight on the penalty term to obtain a zero gap.

Equations

$$z^{\mathrm{IP}}:=\inf \{c^{\top }x+\frac{1}{2}{x}^{\top }Qx:Ax=b,x\in X\},$$

$$X=\{(x_1,x_2)\in {\mathbb{R}}^{n_1}\times {\mathbb{Z}}^{n_2}:Ex\le f\},$$

$$z_{\rho }^{\mathrm{LD}+}:=\underset{\lambda }{\sup }\underset{x\in X}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\lambda }^{\top }(b-Ax)+\rho \psi (b-Ax)\},$$

$$z_{\rho }^{\mathrm{LR}+}(\lambda ):=\underset{x\in X}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\lambda }^{\top }(b-Ax)+\rho \psi (b-Ax)\},$$

$$z_{\rho }^{\mathrm{LD}+}:=\underset{\lambda }{\sup }{z}_{\rho }^{\mathrm{LR}+}(\lambda )=\underset{\lambda }{\sup }\underset{x\in X}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\lambda }^{\top }(b-Ax)+\rho \psi (b-Ax)\}.$$

$$z^{\mathrm{NLP}}:=\inf \{c^{\top }x+\frac{1}{2}{x}^{\top }Qx:Ax=b,Ex\le f,x\in {\mathbb{R}}^{n_1+n_2}\}.$$

$$z^{\mathrm{NLP}}=\inf$$

$$s.t.$$

$$Ex\le f$$

$$=\inf$$

$$s.t.$$

$$Ex\le f$$

$$\ge \inf$$

$$s.t.$$

$$Ex\le f.$$

$$w_{\rho ,ϵ}^{\ast }:=\underset{x,w}{\inf }$$

$$s.t.$$

$$\psi (b-Ax)\le w,$$

$$c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\bar{\lambda }}^{\top }(b-Ax)+\rho w-z_{\rho }^{\mathrm{LR}+}(\bar{\lambda })\le ϵ.$$

$$z_{\rho }^{\mathrm{LR}+}(\bar{\lambda })\ge$$

$$\ge$$

$$=$$

$$w_{\rho ,ϵ}^{\ast }\le$$

$$\le$$

$$0\le w_{\rho }^{\ast }=\underset{ϵ\downarrow 0}{\lim }{w}_{\rho ,ϵ}^{\ast }\le \underset{ϵ\downarrow 0}{\lim }\frac{1}{\rho }(z^{\mathrm{IP}}+ϵ-z^{\mathrm{NLP}})=\frac{1}{\rho }(z^{\mathrm{IP}}-z^{\mathrm{NLP}}).$$

$${\tilde{z}}_{\rho }^{\mathrm{LR}+}(\bar{\lambda }):=\underset{x,w}{\inf }$$

$$s.t.$$

$$\psi (b-Ax)\le w,$$

$$(1-\delta )w_{\rho }^{\ast }\le w\le (1+\delta )w_{\rho }^{\ast }.$$

$$z_{\rho }^{\mathrm{LR}+}(\bar{\lambda })=$$

$$\ge$$

$$s.t.x\in X,$$

$$\psi (b-Ax)\le (1+\delta )w_{\rho }^{\ast },$$

$$\ge$$

$$s.t.x\in X,$$

$$\psi (b-Ax)\le (1+\delta )w_{\rho }^{\ast }.$$

$$c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\bar{\lambda }}^{\top }(b-Ax)+\rho w\ge {\tilde{z}}_{\rho }^{\mathrm{LR}+}(\bar{\lambda })=z_{\rho }^{\mathrm{LR}+}(\bar{\lambda })+{\alpha }_{\rho },$$

$$z_{\rho }^{\mathrm{LD}+}$$

$$\ge \underset{x}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\bar{\lambda }}^{\top }(b-Ax):x\in X,\psi (b-Ax)\le (1+\delta )w_{\rho }^{\ast }\}$$

$$\ge \underset{x}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\bar{\lambda }}^{\top }(b-Ax):x\in X,\psi (b-Ax)\le \frac{2}{\rho }(z^{\mathrm{IP}}-z^{\mathrm{NLP}})\}$$

$$\ge \underset{x}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\bar{\lambda }}^{\top }(b-Ax):x\in X,\Vert b-Ax{\Vert }_{\infty }\le {\kappa }_{\rho }\}(\ge z^{\mathrm{NLP}})$$

$$\underset{x}{\inf }\{c^{\top }x+\frac{1}{2}{x}^{\top }Qx+{\bar{\lambda }}^{\top }(b-Ax):x\in X,\Vert b-Ax{\Vert }_{\infty }\le {\kappa }_{\rho }\}$$

$$=$$

Full text

1 Introduction

We consider the following rational (mixed) integer quadratic programming (MIQP) problem with decision variable $x\in\mathbb{R}^{n}$:

[TABLE]

where the parameters are: a rational symmetric positive semi-definite matrix $Q\in\mathbb{Q}^{n\times n}$, a rational matrix $A\in\mathbb{Q}^{m\times n}$, rational vectors $c\in\mathbb{Q}^{n}$ and $b\in\mathbb{Q}^{m}$, a mixed integer linear set $X$ such that

[TABLE]

where $E\in\mathbb{Q}^{m_{2}\times n}$ is a rational matrix and $f\in\mathbb{Q}^{m_{2}}$ is a rational vector with $n_{1}+n_{2}=n$. We consider dualizing the constraints $Ax=b$.

While for continuous quadratic programming (QP), it is well known that even the classical Lagrangian dual (LD) will reach a zero duality gap and strong duality holds [1], it is not true for MIQP, as the integer variables introduce non-convexity. In fact, LD may have a non-zero duality gap for the problem. Therefore, to close the gap, the idea of penalizing violation of the dualized constraints with a nonlinear penalty gives rise to the well known augmented Lagrangian dual (ALD), which is

[TABLE]

where $\rho>0$ is the penalty weight, and $\psi(\cdot)$ is the penalty function which usually satisfies $\psi(0)=0$ and $\psi(u)>0$ if $u\neq 0$ [15].

Numerous papers have discussed ALD. The paper [14] uses convex quadratic penalty functions for nonconvex programming, [9] discusses the asymptotic zero duality gap and exact penalty representation for mixed integer linear programming (MILP), [5] discusses the optimality conditions for semi-infinite programming, and [4] discusses exact penalization for general augmented Lagrangian.

It should be noted that an exact penalty representation usually requires a much restricted penalty function, like norm functions, see for example [15]. Norm function is used in [6] for exact penalization. The work [10] discusses exact penalty representation using level-bounded augmented functions and [16] considers the penalty function which is almost peak at zero. More recent works like [9, 5] apply sharp Lagrangian to different types of problems.

On the other hand, the size (for example, in binary coding) of the penalty weight is rarely discussed. While there are discussions for the size and computational complexity of MILP [19, 2], QP [18] and MIQP [7], we might be able to utilize their ideas to show the small size of the penalty weight.

In this paper, we significantly generalize the results of [9]. In particular, we

Prove that the duality gap of ALD will asymptotically reach zero under mild conditions as the penalty weight goes to infinity; 2. 2.

Prove that the duality gap will reach zero given that the penalty function is any norm, and the penalty weight is sufficiently large but still finite; 3. 3.

Prove that the size of the penalty weight which attains zero duality gap is polynomially bounded with respect to the problem data.

The paper is organized as follows. In Section 2 we provide definitions and formal statement of main results of the paper. In Section 3 we present several key lemmas useful across the paper. In Section 4 we exhibit properties of ALD as the penalty weight goes to infinity, and show the (asymptotic) zero duality gap for a large class of penalty functions. In Section 5 we show a finite penalty weight whose size is polynomially bounded with respect to the input parameters, for which a zero duality gap is attained.

2 Main Results

In this section, we introduce some definitions and briefly present our main results.

{assumption}

The MIQP Eq. 1 is feasible and the optimal value is bounded.

Definition 2.1**.**

The augmented Lagrangian relaxation is defined as

[TABLE]

where $\psi$ is a penalty function. The augmented Lagrangian dual is defined as

[TABLE]

Definition 2.2**.**

The continuous relaxation of Eq. 1 is denoted as $z^{\mathrm{NLP}}$

[TABLE]

Remark 2.3**.**

*We use $\bar{\lambda}$ to denote the optimal dual variables (of $z^{\mathrm{NLP}}$) for the constraints $Ax=b$ and $\bar{\lambda}_{E}$ to denote the optimal dual variables for $Ex\leq f$. The existence of $\bar{\lambda}$ and $\bar{\lambda}_{E}$ is guaranteed by the boundedness of the continuous relaxation, which is given by Lemma 3.1. *

Remark 2.4**.**

*For any $\rho$, $\lambda$, we have $z_{\rho}^{\mathrm{LR+}}(\lambda)\leq z_{\rho}^{\mathrm{LD+}}\leq z^{\mathrm{IP}}$. Moreover, we have $z^{\mathrm{NLP}}=\inf\{c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax):Ex\leq f,x\in\mathbb{R}^{n_{1}+n_{2}}\}\leq z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})\leq z_{\rho}^{\mathrm{LD+}}\leq z^{\mathrm{IP}}$. *

Definition 2.5**.**

*For a finite set of vectors $T=\{t_{1},t_{2},...,t_{k}\}$, $\mathrm{conv}(T)$, $\mathrm{cone}(T)$ and $\mathrm{int.cone}(T)$ are the convex hull, conical hull and integral conical hull of $T$, respectively. Here, $\mathrm{int.cone}(T):=\{\sum_{i=1}^{k}\mu_{i}t_{i}:\mu_{i}\in\mathbb{Z}_{+}\}$. *

Definition 2.6**.**

*For any subset $T$ of a metric space, its diameter $\rm{\mathrm{diam}}$ is defined as ${\rm{\mathrm{diam}}}(T)=\sup_{a,b\in T}\|a-b\|$, where $\|\cdot\|$ is the metric associated with the space. *

Definition 2.7** ([7]).**

*Given an object $\mathcal{O}$ and another object $f(\mathcal{O})$ which is a function of it, we say that $f(\mathcal{O})$ has $\mathcal{O}$-small complexity, if the size (in standard binary encoding) of $f(\mathcal{O})$ is bounded above by a a polynomial function of the size of $\mathcal{O}$. *

Definition 2.8**.**

*We use $\mathcal{F}$ to denote all input parameters of Eq. 1 including $E$, $f$, $c$, $Q$, $A$ and $b$. In addition, any object $q$ which is a function of $\mathcal{F}$ is said to have small complexity, if $q$ has $\mathcal{F}$-small complexity. *

Below we present the main theorems of the paper.

Theorem 2.9** (Asymptotic Zero Duality Gap).**

*Assume $\psi$ is proper, nonnegative, lower-semicontinuous and level-bounded, that is: $\psi(0)=0$; $\psi(u)>0$ for all $u\neq 0$; $\lim_{\delta\downarrow 0}{\rm{diam}}\{u:\psi(u)\leq\delta\}=0$; ${\rm{diam}}\{u:\psi(u)\leq\delta\}<\infty$ for all $\delta>0$. We have $\sup_{\rho>0}z^{\mathrm{LD+}}_{\rho}=z^{\mathrm{IP}}$. *

Theorem 2.10** (Exact Penalty Representation).**

Suppose $\psi(\cdot)$ is any norm.

1. (a)

There exists a $\rho^{*}$ of small complexity, such that $z^{\mathrm{LD+}}_{\rho^{*}}=z^{\mathrm{IP}}$. 2. (b)

Moreover, for all $\lambda$, there exists a finite $\rho^{*}(\lambda)$ of $\mathcal{F},\lambda$-small complexity, such that $z^{\mathrm{LR+}}_{\rho^{*}}(\lambda)=z^{\mathrm{IP}}$.

We provide a flowchart that depicts how the preliminary results proved in Section 3 are put together to prove Theorem 2.9.

Lemma 3.1

Equivalence of Boundedness of MIQP and its Continuous Relaxation

Proposition 4.1

Approximation of the Penalty Term

Lemma 4.3

Equivalent Form of $z^{\mathrm{LR+}}_{\rho}(\bar{\lambda})$

Lemma 3.4

Decomposition of Rational Mixed Integer Polyhedron

Lemma 3.6

Bounded Region with Small Complexity

Lemma 4.6

Adding a Uniform Bound on $x$ without Changing the Value

Theorem 2.9

Asymptotic Zero Duality Gap

Another flowchart is provided respect to the proof of Theorem 2.10.

Lemma 3.1

Equivalence of Boundedness of MIQP and its Continuous Relaxation

Theorem 5.1

A Sufficient Condition for Exact Penalty

Theorem 2.9

Asymptotic Zero Duality Gap

Lemma 3.4

Decomposition of Rational Mixed Integer Polyhedron

Lemma 3.6

Bounded Region with Small Complexity

Lemma 5.5

A Uniform Bound on $x$ Independent of $\rho$

Theorem 5.3

Exact Penalty Representation for $L^{\infty}$ Norm

Theorem 2.10

Exact Penalty Representation for Any Norm

3 Preliminary Results

Several useful lemmas are presented in this section.

Lemma 3.1** (Equivalence of Boundedness of MIQP and its Continuous Relaxation).**

Suppose the MIQP is feasible (i.e. $z^{\mathrm{IP}}<+\infty$). Then the following three conditions are equivalent:

$z^{\mathrm{NLP}}$* is bounded.* 2. 2.

$\inf\{c^{\top}x|Ax=0,Ex\leq 0,Qx=0\}$* is bounded.* 3. 3.

$z^{\mathrm{IP}}$* is bounded.*

Proof 3.2**.**

$1\Rightarrow 3$* is obvious.*

We first show $3\Rightarrow 2$, or equivalently $\neg 2\Rightarrow\neg 3$. Note that the problem in 2 is always feasible. Assuming $\neg 2$, the problem $\{c^{\top}x\leq-1,Ax=0,Ex\leq 0,Qx=0\}$ is now feasible and there exists a rational solution since the problem is rational. Denote such a rational solution as $r$ and without loss of generality, we assume that $r$ is integral since we can scale $r$ with a positive coefficient.

Now select any feasible solution for 3, as $x$. Then we know that $x+tr$ is still feasible for 3 for any $t\in\mathbb{Z}_{+}$. In addition, $c^{\top}(x+tr)+\frac{1}{2}(x+tr)^{\top}Q(x+tr)=c^{\top}x+\frac{1}{2}x^{\top}Qx+tc^{\top}r\to-\infty$ as $t\to+\infty$. Therefore, we have 3 is unbounded, i.e. $3\Rightarrow 2$.

Next we show that $2\Rightarrow 1$. Suppose that 2 holds. From Farkas Lemma, we know that $\exists\lambda_{E}\leq 0,\lambda_{A},\lambda_{Q}$, such that $\lambda_{E}^{\top}E+\lambda_{A}^{\top}A+\lambda_{Q}^{\top}Q=c^{\top}$. Now considering the NLP

[TABLE]

*To show that $\lambda_{Q}^{\top}Qx+\frac{1}{2}x^{\top}Qx$ is bounded, we first write down the orthogonal decomposition of $Q$ as $R^{\top}\Lambda R$ where $R$ is orthogonal and $\Lambda$ is diagonal. Therefore, $\lambda_{Q}^{\top}Qx+\frac{1}{2}x^{\top}Qx=\sum_{i}\Lambda_{ii}(\lambda_{Q}^{\top}r_{i}+\frac{1}{2}x^{\top}r_{i})x^{\top}r_{i}\geq\sum_{i}-\Lambda_{ii}(\lambda_{Q}^{\top}r_{i})^{2}/2=-\lambda_{Q}^{\top}Q\lambda_{Q}/2$, where $r_{i}$ is the $i$-th row of the orthogonal matrix $R$. Note that the bound is attainable if we take $x=-\lambda_{Q}$. Therefore, we arrive at 1, i.e. $z^{\mathrm{NLP}}$ is bounded. *

Remark 3.3**.**

*We note here that we are able to prove that the boundedness of the nonlinear integer problem implies boundedness of its continuous relaxation, using the fact that the data is rational. This is very similar to the Fundamental theorem of Integer Programming [11]. Note that other similar results may be proven under different assumptions such as existence of integer point in the interior of continuous relaxation, see [8, 13]. Also see [12]. *

Lemma 3.4** (Decomposition of Rational Mixed Integer Polyhedron).**

Given a rational positive semidefinite matrix $Q$, any rational mixed integer polyhedron $P\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})=\{x:Cx\leq d\}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})$ can be decomposed (with respect to $Q$) as $\cup_{i}(P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})+\mathrm{int.cone}(R_{i}))$ satisfying the following properties:

1. (a)

Each $P_{i}$ is a rational polytope. 2. (b)

Each $\mathrm{cone}(R_{i})$ is a rational, simple and pointed cone. 3. (c)

For every $\mathrm{cone}(R_{i})$, if a face $C^{\prime}$ satisfies that $\exists x\in C^{\prime}\backslash\{0\}$, $x^{\top}Qx=0$, then there exists an extreme ray $v$ of $C^{\prime}$ with $v^{\top}Qv=0$. 4. (d)

Each polytope $P_{i}$ and each vector in $R_{i}$ has $P,Q$-small complexity.

Proof 3.5**.**

This lemma is a direct outcome of [7, Proposition 1, Proposition 2, Lemma 2].

First, if $P$ is not pointed, we can decompose $P$ into at most $2^{n_{1}+n_{2}}$ pointed rational mixed integer polyhedron by separating $x_{k}\leq 0$ and $x_{k}\geq 0$ for all $k$. Therefore, we simply assume $P$ is pointed henceforth.

Next, using [7, Proposition 1], we can decompose $P$ as $P=\cup_{i,K^{1}\in\mathcal{K}^{1}}(P^{1}_{i}+\mathrm{cone}(R^{1}_{K^{1}}))$, while conditions (a), (b), (d) are met.

Later, using [7, Lemma 2], we are able to decompose $\mathrm{cone}(R_{k})$ into a union of rational, simple and pointed cones, which satisfies condition (c) and maintains (a), (b), (d). Therefore, $P=\cup_{i,K^{2}\in\mathcal{K}^{2}}(P^{1}_{i}+\mathrm{cone}(R^{2}_{K^{2}}))$.

*Finally, we use [7, Proposition 2] and decompose $(P^{1}_{i}+\mathrm{cone}(R^{2}_{K^{2}}))\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})$ into a mixed integer rational polytope plus an integer cone, which completes the proof. *

Lemma 3.6** (Bounded Region with Small Complexity [7]).**

Let $P\subseteq R^{n}$ be a polytope and $R\subseteq R^{n}$ be a finite set of vectors. Given a rational positive semidefinite matrix $Q$ of small complexity, suppose $P+\mathrm{cone}(R)$ satisfies the following properties:

1. (a)

$P$* is a rational polytope.* 2. (b)

$\mathrm{cone}(R)$* is a rational, simple and pointed cone.* 3. (c)

$\forall x\in\mathrm{cone}(R)\backslash\{0\}$, $x^{\top}Qx>0$. 4. (d)

$P$* and each vector in $R$ has small complexity.*

*Then, for any $\eta\in\mathbb{R}^{n},\mu\in\mathbb{R}$ of small complexity, there exists $M$ of small complexity such that $\{x\in P+\mathrm{cone}(R):\frac{1}{2}x^{\top}Qx+\eta^{\top}x\leq\mu\}\subset\{x:\|x\|\leq M\}$. In addition, such $M$ exists for any norm. *

Remark 3.7**.**

For any rational mixed integer polyhedron, Lemma 3.4 provides a decomposition with respect to $Q$, while maintaining a small complexity. In addition, for any part of the decomposition, if no extreme ray $v$ has $v^{\top}Qv=0$ then no ray has $x^{\top}Qx=0$ (i.e. $x^{\top}Qx>0$ for any ray).

Lemma 3.6* shows that under the conditions that $x^{\top}Qx>0$ for any ray, the optimal solution of the optimization problem $\{\min\frac{1}{2}x^{\top}Qx+\eta^{\top}x:x\in P+\mathrm{cone}(R)\}$ will have small complexity. In the lemma, this property is presented in the form of a feasibility problem.*

*The two lemmas will be needed for proving bounds in our proofs of theorems. *

4 Asymptotic Zero Duality Gap

In this section, we show that under mild conditions on the penalty function the ALD duality gap vanishes as the penalty weight $\rho$ goes to infinity. {assumption}[Conditions for Asymptotic Zero Duality Gap]

We assume $\psi$ is proper, nonnegative, lower-semicontinuous and level-bounded, that is: $\psi(0)=0$; $\psi(u)>0$ for all $u\neq 0$; $\lim_{\delta\downarrow 0}{\rm{diam}}\{u:\psi(u)\leq\delta\}=0$; ${\rm{diam}}\{u:\psi(u)\leq\delta\}<\infty$ for all $\delta>0$.

Proposition 4.1** (Approximation of the Penalty Term).**

For given $\rho>0$ and $\epsilon>0$, define $w_{\rho,\epsilon}^{*}$ as

[TABLE]

*Then, the limit $w^{*}_{\rho}:=\lim_{\epsilon\downarrow 0}w_{\rho,\epsilon}^{*}$ exists and $\lim_{\rho\to+\infty}w^{*}_{\rho}=0$. *

Proof 4.2**.**

First we need show that the problem Eq. 2 is well-defined, i.e is feasible and bounded. As a first step we show that $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ is finite. Observe that:

[TABLE]

and the boundedness of $z^{\mathrm{NLP}}$ is given by Lemma 3.1.

From the feasibility of the original problem we know that there exists an $x$ feasible for $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$. Therefore, we are able to find $x\in X$ such that $c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax)+\psi(b-Ax)\leq\epsilon+z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$, which means $(x,w=\psi(b-Ax))$ is feasible for Eq. 2. We also have $w^{*}_{\rho,\epsilon}\geq 0$ from the non-negativity of $\psi$. Thus, Eq. 2 is feasible and bounded.

In addition, we have $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})\leq z^{\mathrm{IP}}$ and for $x$ satisfying $Ex\leq f$ we have that $c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax)\geq\inf\{c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax)+\bar{\lambda}^{\top}_{E}(f-Ex)\}=z^{\mathrm{NLP}}$. Therefore,

[TABLE]

By taking $\epsilon\downarrow 0$ we have

[TABLE]

In addition, as $\epsilon\downarrow 0$ the feasible region of Eq. 2 becomes smaller, which indicates that $w_{\rho,\epsilon}^{*}$ is non-decreasing. Therefore $w_{\rho}^{*}=\lim_{\epsilon\downarrow 0}w_{\rho,\epsilon}^{*}$ exists.

*By taking $\rho\to+\infty$ we therefore obtain $\lim_{\rho\to+\infty}w^{*}_{\rho}=0$. *

Lemma 4.3** (Equivalent Form of $z^{\mathrm{LR+}}_{\rho}(\bar{\lambda})$).**

Consider $w^{*}_{\rho}$ as in Proposition 4.1 and define $\tilde{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ as

[TABLE]

Then, for any $\delta\in(0,1)$,

[TABLE]

Proof 4.4**.**

Note that the definition of $\tilde{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ is the same as that of $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ except for the additional constraint $(1-\delta)w^{*}_{\rho}\leq w\leq(1+\delta)w^{*}_{\rho}$. Suppose $\alpha_{\rho}:=\tilde{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})-z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})>0$ by contradiction. Then, for all $(x,w)$ feasible to Eq. 4 we have

[TABLE]

*which implies $(x,w)$ is infeasible for Eq. 2 if $\epsilon<\alpha_{\rho}$. Hence, $w^{*}_{\rho,\epsilon}\notin((1-\delta)w_{\rho}^{*},(1+\delta)w_{\rho}^{*}$), a contradiction. Therefore $\tilde{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})=z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ and the inequalities Eq. 5 are straightforward to verify. *

We are now ready to present the asymptotic zero duality gap. See 2.9

Proof 4.5**.**

$z^{\mathrm{LD+}}_{\rho}$* does not decrease as $\rho$ increases. Therefore, it is then sufficient to show that $\sup_{\rho\geq 1}z^{\mathrm{LD+}}_{\rho}=z^{\mathrm{IP}}$ under the assumption.*

Let $\delta\in(0,1)$, and we have

[TABLE]

where $\kappa_{\rho}:=\mathrm{diam}\{u:\psi(u)\leq\frac{2}{\rho}(z^{\mathrm{IP}}-z^{\mathrm{NLP}})\}$ which is obviously non-increasing with respect to $\rho$. Eq. 6a is guaranteed by Lemma 4.3. Eq. 6b is valid from Eq. 3 and Eq. 6c comes from the level-boundedness of Section 4.

We will need the following lemma that provides a uniform bound $M$ on Eq. 6c for $x$ independent of $\rho$.

Lemma 4.6** (Adding a Uniform Bound on $x$ without Changing the Value).**

Under the assumption that $\rho\geq 1$, $\exists M>0$ independent of $\rho$, such that

[TABLE]

A proof of Lemma 4.6 is provided later. From Lemma 4.6 we have $z_{\rho}^{\mathrm{LD+}}\geq\min_{x}\{c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax):x\in X,\|b-Ax\|_{\infty}\leq\kappa_{\rho},\|x\|_{\infty}\leq M\}$. By taking $\rho\to+\infty$, we get

[TABLE]

*where Eq. 7a follows from the finiteness of $x_{2}$ under $\|x_{2}\|_{\infty}\leq M$, Eq. 7b follows from the lower semi-continuity of the continuous quadratic programming [1, Proposition 6.5.2] and Eq. 7c holds by Section 4. Note that for any $\rho,\lambda$, $z_{\rho}^{\mathrm{LR+}}(\lambda)\leq z_{\rho}^{\mathrm{LD+}}\leq z^{\mathrm{IP}}$, and thus $\lim_{\rho\to+\infty}z_{\rho}^{\mathrm{LD+}}=z^{\mathrm{IP}}$. *

Note that by proving the theorem we also show that $\lim_{\rho\to+\infty}z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})=z^{\mathrm{IP}}$ from the non-decreasing of $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ with respect to $\rho$.

We now complete the proof by proving Lemma 4.6

Proof 4.7** (Proof of Lemma 4.6).**

Note that $\|b-Ax\|_{\infty}\leq\kappa_{\rho}$ can be written as linear constraints. Hence, apply Lemma 3.4 to the feasible region for $\rho=1$ of Eq. 6b and we get a decomposition $\cup_{i}(P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})+\mathrm{int.cone}(R_{i}))$ with the properties listed in the lemma. Note that for all $r\in R_{i}$ we have $Ar=0$ from the constraints $\|b-Ax\|_{\infty}\leq\kappa_{1}$. Therefore, the feasible region for any $\rho\geq 1$ can be written as

[TABLE]

Now consider the problem $\inf_{x}\{c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax):x\in P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})\cap\{x:\|b-Ax\|_{\infty}\leq\kappa_{\rho}\})+\mathrm{int.cone}(R_{i})\}$. If there exists $r\in R_{i}$ such that $r^{\top}Qr=0$ (i.e. $Qr=0$), then the feasible region can be rewritten as $P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}}\cap\{x:\|b-Ax\|_{\infty}\leq\kappa_{\rho}\})+\mathrm{int.cone}(R_{i}\backslash\{r\})+\{\mu r:\mu\in\mathbb{Z}_{+}\}$.

We can use $y+\mu r$ such that $y\in P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}}\cap\{x:\|b-Ax\|_{\infty}\leq\kappa_{\rho}\})+\mathrm{int.cone}(R_{i}\backslash\{r\})$ and $\mu\in\mathbb{Z}_{+}$ to represent $x$. The problem is therefore $\inf_{y,\mu}\{(c^{\top}-\bar{\lambda}^{\top}A)\mu+c^{\top}y+\frac{1}{2}y^{\top}Qy+\bar{\lambda}^{\top}(b-Ay):y\in P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}}\cap\{x:\|b-Ax\|_{\infty}\leq\kappa_{\rho}\})+\mathrm{int.cone}(R_{i}\backslash\{r\}),\mu\in\mathbb{Z}_{+}\}$. Optimize the problem over $\mu$ and we get $\mu=0$ an optimal solution (or the problem is unbounded, contrary to Eq. 6b). Therefore, we can refine the feasible region by omitting all $r\in R_{i}$ such that $Qr=0$. Denote the set after the process as $R_{i}^{J}$. Note that this process is independent of the value of $\rho$, and hence we have

[TABLE]

In addition, from (c) of Lemma 3.4, for all $x\in\mathrm{cone}(R_{i}^{J})\backslash\{0\}$, we have $x^{\top}Qx>0$. Let

[TABLE]

Note that the definition on $V_{i}$ is independent of $\rho$. Note that $(P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}})\cap\{x:\|b-Ax\|_{\infty}\leq\kappa_{\rho}\})+\mathrm{int.cone}(R_{i}^{J}))\subseteq(P_{i}+\mathrm{cone}(R_{i}^{J}))$ and we have

[TABLE]

Using Lemma 3.6 we have that there exists $M_{i}>0$ such that $V_{i}\in\{x:\|x\|_{\infty}\leq M_{i}\}$. Take $M=\max\{M_{i}\}$, which is independent of $\rho$, and we have

[TABLE]

While $\inf_{x}\{c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax)\ \mathrm{s.t.}\ x\in X,\|b-Ax\|_{\infty}\leq\kappa_{\rho},\|x\|_{\infty}\leq M\}\geq z^{\mathrm{NLP}}$ is bounded and the values of $x_{2}$ here is finite, we can therefore replace $\inf$ by $\min$.

Therefore,

[TABLE]

Since it is obvious that

[TABLE]

*thus equality holds and the proof is completed. *

5 Exact Penalty Representation

In this section, we will discuss conditions for an exact penalty representation. To begin with, a sufficient condition is given. We later prove the sufficiency of using norm as the penalty function for an exact penalty, while noting that a norm function always satisfies Section 4.

Theorem 5.1** (A Sufficient Condition for Exact Penalty).**

Under Section 2 (MIQP is feasible and the optimal value is bounded), if there exists $\delta$, such that

[TABLE]

*and $\psi(0)=0$, then there exists a finite $\rho^{*}$ such that $z^{\mathrm{LR+}}_{\rho^{*}}(\bar{\lambda})=z^{\mathrm{IP}}$, which also gives $z^{\mathrm{LD+}}_{\rho^{*}}=z^{\mathrm{IP}}$. *

Proof 5.2**.**

Under Section 2, using Lemma 3.1, we have $z^{\mathrm{NLP}}$ is bounded. Thus, choose a feasible point $\tilde{x}$ for the MIQP and set

[TABLE]

we next show that $\rho^{*}$ satisfies our requirements.

First of all, as $z^{\mathrm{NLP}}$ bounded and $c^{\top}\tilde{x}+\frac{1}{2}\tilde{x}^{\top}Q\tilde{x}\geq z^{\mathrm{IP}}$, we have $\rho^{*}\in[0,+\infty)$. Clearly $z^{\mathrm{LR+}}_{\rho^{*}}(\bar{\lambda})\leq z^{\mathrm{IP}}$. We next show that $z^{\mathrm{LR+}}_{\rho^{*}}(\bar{\lambda})\geq z^{\mathrm{IP}}$.

For $x\in X$ with $Ax=b$, we have

[TABLE]

For $x\in X$ with $Ax\neq b$, we have

[TABLE]

from the strong duality results for QP. Thus,

[TABLE]

*Therefore, we have for $x\in X$, $c^{\top}x+\frac{1}{2}x^{\top}Qx+\bar{\lambda}^{\top}(b-Ax)+\rho^{*}\psi(b-Ax)\geq z^{\mathrm{IP}}$ and thus $z^{\mathrm{LR+}}_{\rho^{*}}(\bar{\lambda})=z^{\mathrm{IP}}$. *

We now present the exact penalty results for $\psi(\cdot)=\|\cdot\|_{\infty}$.

Theorem 5.3** (Exact Penalty Representation for $L^{\infty}$-Norm).**

*Assuming $\psi(\cdot)=\|\cdot\|_{\infty}$, there exists a finite $\rho^{*}(\bar{\lambda})$ of small complexity, such that $z_{\rho^{*}}^{\mathrm{LR+}}(\bar{\lambda})=z^{\mathrm{IP}}$. *

Proof 5.4**.**

It is sufficient to find a finite $\rho^{*}(\bar{\lambda})$ polynomially bounded, such that $z_{\rho^{*}}^{\mathrm{LR+}}(\bar{\lambda})\geq z^{\mathrm{IP}}$. Since $z^{\mathrm{LR+}}_{\rho}(\bar{\lambda})$ is non-decreasing with $\rho$ increasing, without loss of generality, we only consider $\rho\geq 1$. In addition, from [7, Theorem 4], $z^{\mathrm{IP}}$, $z^{\mathrm{NLP}}$ and $\bar{\lambda}$ have $\mathcal{F}$-small complexity.

The constraints $\|b-Ax\|_{\infty}\leq w$ can be written as $-\mathbf{1}w\leq b-Ax\leq\mathbf{1}w$. Therefore,

[TABLE]

The following lemma shows a uniform bound $M$ of small complexity can be put on $x$ independent of $\rho$.

Lemma 5.5** (A Uniform Bound on $x$ Independent of $\rho$).**

Under the assumption that $\rho\geq 1$, there exists $M>0$ independent of $\rho$ and of small complexity, such that

[TABLE]

A proof of Lemma 5.5 is provided later. We next rewrite $x=(x_{1},x_{2})$ and separate $A,E,c$ respectively. We also rewrite

[TABLE]

Note that $Q^{(11)}$ is also positive semi-definite. Therefore, the problem can be rewrite as

[TABLE]

Denote $V=\{v\in\mathbb{Z}^{n_{2}}:\|v\|_{\infty}\leq M\}$. In addition, we use $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})$ to denote $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ while fixing $x_{2}$. Therefore, $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})=\min_{x_{2}\in V}z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})$. Note that $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})$ is still non-decreasing with respect to $\rho$. Therefore, from Theorem 2.9 we have $z^{\mathrm{IP}}=\lim_{\rho\to+\infty}\min_{x_{2}\in V}z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})=\min_{x_{2}\in V}\lim_{\rho\to+\infty}z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})$, or equivalently we have that $\lim_{\rho\to+\infty}z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})\geq z^{\mathrm{IP}}$.

For arbitrary $x_{2}\in V$, the dual problem of Eq. 10 (with respect to $x_{1},w$) is therefore

[TABLE]

Note that the problem $\inf_{x_{1},w}$ is bounded if and only if $(y_{1}+y_{2})^{\top}\mathbf{1}=\rho$ and there exists $\nu$, s.t. $c_{1}^{\top}-\bar{\lambda}^{\top}A_{1}+x_{2}^{\top}Q^{(21)}+(y_{1}-y_{2})^{\top}A_{1}+y_{3}^{\top}E_{1}+(y_{4}-y_{5})^{\top}=\nu^{\top}Q^{(11)}$. Therefore the problem is

[TABLE]

By strong duality we have $z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})=z^{\text{DRD+}}_{\rho}(\bar{\lambda},x_{2})$. Therefore, we have $\lim_{\rho\to+\infty}z_{\rho}^{\text{DRD+}}(\bar{\lambda},x_{2})=\lim_{\rho\to+\infty}z_{\rho}^{\mathrm{LR+}}(\bar{\lambda},x_{2})\geq z^{\mathrm{IP}}$. Now consider the following problem with respect to $(\xi,y,\nu,\rho)$:

[TABLE]

The above problem is a quadratically constrained quadratic programming (QCQP) with convex constraints and affine objective function. The existence of an optimal solution is guaranteed by the finiteness and feasibility of the problem [1]. In addition, as $\lim_{\rho\to+\infty}z_{\rho}^{\text{DRD+}}(\bar{\lambda},x_{2})\geq z^{\mathrm{IP}}$ there exists a sequence of $(\xi^{k},y^{k},\nu^{k},\rho^{k})$ feasible to the problem such that $\xi^{k}\to 0$. Therefore, the optimal value is [math] and an optimal solution $(0,y^{*},\nu^{*},\rho^{*})$ exists, which guarantees the feasibility of the following problem with respect to $(y,\nu,\rho)$:

[TABLE]

Similarly, the finiteness and feasibility of the problem guarantees the existence of the optimal solution. Therefore, $\rho^{*}(x_{2})$ is well defined and from [7, Theorem 4], $\rho^{*}(x_{2})$ has small complexity. In addition, we have $z^{\text{DRD+}}_{\rho^{*}(x_{2})}(\bar{\lambda},x_{2})\geq z^{\mathrm{IP}}$.

Now let $\rho^{*}=\max_{x_{2}\in V}\rho^{*}(x_{2})$ of small complexity, we have $z^{\text{DRD+}}_{\rho^{*}}(\bar{\lambda},x_{2})\geq z^{\text{DRD+}}_{\rho^{*}(x_{2})}(\bar{\lambda},x_{2})\geq z^{\mathrm{IP}}$ for all $x_{2}\in V$. Hence, $z^{\mathrm{LR+}}_{\rho^{*}}(\bar{\lambda})=\min_{x_{2}\in V}z^{\mathrm{LR+}}_{\rho^{*}}(\bar{\lambda},x_{2})=\min_{x_{2}\in V}z^{\text{DRD+}}_{\rho^{*}}(\bar{\lambda},x_{2})\geq z^{\mathrm{IP}}$.

*Note that for any $\rho,\lambda$, $z_{\rho}^{\mathrm{LR+}}(\lambda)\leq z_{\rho}^{\mathrm{LD+}}\leq z^{\mathrm{IP}}$, and thus $z_{\rho^{*}}^{\mathrm{LR+}}(\bar{\lambda})=z_{\rho^{*}}^{\mathrm{LD+}}=z^{\mathrm{IP}}$. *

We next complete the proof by proving Lemma 5.5.

Proof 5.6** (Proof of Lemma 5.5).**

Consider for $\rho\geq 1$, $w_{M}:=2(z^{\mathrm{IP}}-z^{\mathrm{NLP}})\geq 2w^{*}_{\rho}$, where the inequality follows Eq. 3, and define

[TABLE]

Denote the feasible region for this problem as $P$ and the feasible region for the original problem Eq. 8 as $P^{o}$. Clearly, $P\subseteq P^{o}$ and thus $\hat{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})\geq z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$. Similarly as $P$ is larger than the feasible region of Eq. 4, we have $\hat{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})\leq\tilde{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})=z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$ and thus $\hat{z}_{\rho}^{\mathrm{LR+}}(\bar{\lambda})=z_{\rho}^{\mathrm{LR+}}(\bar{\lambda})$.

Now apply Lemma 3.4 to $P$ and we get a decomposition $P=\cup_{i}(P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}}\times\mathbb{R})+\mathrm{int.cone}(R_{i}))$ with the properties listed in the lemma. Note that the decomposition applies to all $\rho$.

Note that the problem is bounded from the boundedness of $z_{\rho}^{\mathrm{LR+}}$ and for all $r\in R_{i}$ the $w$-component is [math] (from the constraints $w\leq w_{M}$, so we can omit the $w$-component for any vector in $R_{i}$, or simply denote it as $R_{i}\times\{0\}$.

Similar to the proof to Lemma 4.6, when we solve the problem $\inf_{x,w}\{(c^{\top}-\bar{\lambda}^{\top}A)x+\frac{1}{2}x^{\top}Qx+\rho w+\bar{\lambda}^{\top}b:(x,w)\in P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}}\times\mathbb{R})+\mathrm{int.cone}(R_{i})\times\{0\}\}$, if there exists $r\in R_{i}$ such that $r^{\top}Qr=0$ (i.e. $Qr=0$), the feasible region can be decomposed as as $P_{i}\cap(\mathbb{R}^{n_{1}}\times\mathbb{Z}^{n_{2}}\times\mathbb{R})+\mathrm{int.cone}(R_{i}\backslash\{r\})\times\{0\}+\{\mu r:\mu\in\mathbb{Z}_{+}\}\times\{0\}$. Optimize the problem over $\mu$ and we get $\mu=0$ an optimal solution (otherwise the problem will be unbounded). Therefore, we can refine the feasible region by omitting all $r\in R_{i}$ such that $Qr=0$. Denote the set after the process as $R_{i}^{J}$. Note that this process is independent of $\rho$, and hence we have

[TABLE]

Now, from (c) of Lemma 3.4, for all $x\in\mathrm{cone}(R_{i}^{J})\backslash\{0\}$, we have $x^{\top}Qx>0$.Let

[TABLE]

Note that the definition on $V_{i}$ is independent of $\rho$. Therefore, as $\rho\geq 1$ we have

[TABLE]

From Lemma 3.6, there exists $M_{i}>0$, such that $V_{i}\subseteq\{(x,w):\|(x,w)\|_{\infty}\leq M_{i}\}$ and $M_{i}$ has small complexity. Therefore, $V_{i}\subset\{(x,w):\|x\|_{\infty}\leq M_{i}\}$. Let $M=\max\{M_{i}\}$ (which is again independent of $\rho$ and has small complexity) and we have

[TABLE]

Since

[TABLE]

*equality holds and the proof is completed. *

Next, we will generalize the result to any norm penalty and any dual variable. See 2.10

Proof 5.7**.**

Denote the $\rho^{*}(\bar{\lambda})$ in Theorem 5.3 as $\hat{\rho}$ to avoid confusion.

As $\psi(\cdot)$ is a norm function, there exists $\gamma\in[1,+\infty)$ such that $\gamma\|\cdot\|_{\infty}\geq\psi(\cdot)\geq\|\cdot\|_{\infty}/\gamma$. Without loss of generality, we round up $\gamma$ to a closest integer, which is still a constant decided only by $\|\cdot\|_{\infty}$ and $\psi(\cdot)$. Therefore, by letting $\rho^{*}(\bar{\lambda})=\gamma\hat{\rho}$, which still has small complexity, we have

[TABLE]

where the last equation comes from Theorem 5.3. Along with $z_{\rho}^{\mathrm{LR+}}(\lambda)\leq z_{\rho}^{\mathrm{LD+}}\leq z^{\mathrm{IP}}$, we have $z^{\mathrm{LR+}}_{\rho^{*}(\bar{\lambda})}(\bar{\lambda})=z^{\mathrm{LD+}}_{\rho^{*}(\bar{\lambda})}=z^{\mathrm{IP}}$ and (a) is proven. Now it only remains to show that we can replace $\bar{\lambda}$ by any dual vector $\tilde{\lambda}\in\mathbb{R}^{m}$.

From Cauchy-Schwarz inequality, we have

[TABLE]

Again, applying the property of the norm, there exists $\eta\in[1,+\infty)\cap\mathbb{Z}_{+}$ decided only by $\|\cdot\|_{2}$ and $\psi(\cdot)$, such that $\eta\|\cdot\|_{2}\geq\psi(\cdot)\geq\|\cdot\|_{2}/\eta$, and we have

[TABLE]

By setting $\rho^{*}(\tilde{\lambda})=\lceil\rho^{*}(\bar{\lambda})+\eta\|\tilde{\lambda}-\bar{\lambda}\|_{2}\rceil$, which has $\mathcal{F},\tilde{\lambda}$-small complexity, we have

[TABLE]

*Therefore, along with $z_{\rho}^{\mathrm{LR+}}(\lambda)\leq z^{\mathrm{IP}}$, we have $z^{\mathrm{LR+}}_{\rho^{*}(\tilde{\lambda})}(\tilde{\lambda})=z^{\mathrm{IP}}$. *

Remark 5.8**.**

*The results also apply to MILP, which yield that exact penalty weight $\rho^{*}$ (which is detailedly discussed in [9]) also has $\mathcal{F}$-small complexity. *

6 Conclusions

In this paper, we investigate ALD for MIQP. We prove that an asymptotic zero duality gap is reachable as the penalty weight goes to infinity, under some mild conditions (Section 4) on the penalty function. We also show that a finite penalty weight is enough for an exact penalty when we use any norm as the penalty function. Moreover, we prove that a penalty weight of polynomial size is enough to give an exact penalty representative.

By dualizing and penalizing the difficult constraints using ALD, we can convert the problem to one with an easy feasible region, while maintains the optimality of the original optimal points. However, by introducing a penalty term in the objective function, it might become more difficult to deal with. In addition, as ALD does not deal with integer constraints, the problem is still far from convex.

A special case where the easy constraints are separable, leads us to consider the alternating direction method of multipliers (ADMM) [3] and relative update schemes, which are proposed to solve convex problems separably. However, for mixed integer problems, such methods are mainly heuristic, like [17] for MIQP based on ADMM. Future development of separable exact algorithms utilizing the strong duality results and solving general non-convex problems is a potential direction of research.