ADMM for Multiaffine Constrained Optimization

TL;DR

This paper extends ADMM to solve multiaffine constrained problems with convergence guarantees under weak assumptions, applicable to nonconvex, nonsmooth, and multi-block scenarios, with practical examples.

Contribution

It provides the first convergence analysis of ADMM for multiaffine constraints with weak assumptions, including nonconvex and nonsmooth problems, and demonstrates broad applicability.

Findings

ADMM converges to stationary points under certain conditions.

Applicable to nonconvex, nonsmooth, multi-block problems.

Subproblems have closed-form solutions in examples.

Abstract

We expand the scope of the alternating direction method of multipliers (ADMM). Specifically, we show that ADMM, when employed to solve problems with multiaffine constraints that satisfy certain verifiable assumptions, converges to the set of constrained stationary points if the penalty parameter in the augmented Lagrangian is sufficiently large. When the Kurdyka-\L{}ojasiewicz (K-\L{}) property holds, this is strengthened to convergence to a single constrained stationary point. Our analysis applies under assumptions that we have endeavored to make as weak as possible. It applies to problems that involve nonconvex and/or nonsmooth objective terms, in addition to the multiaffine constraints that can involve multiple (three or more) blocks of variables. To illustrate the applicability of our results, we describe examples including nonnegative matrix factorization, sparse learning, risk parity portfolio selection, nonconvex formulations of convex problems, and neural network training. In each case, our ADMM approach encounters only subproblems that have closed-form solutions.

Figures

Tables

Table 1. Table 1 . Contents of Assumption   1

	Assumption 1
$𝒵$-block operator $Q$	$\mathrm{Im}\left(Q\right)\supseteq \mathrm{Im}\left(A\right)$
Total objective function $\varphi$	$\varphi$ coercive on feasible region
	$\varphi$ splits as $f\left(𝒳\right)+\psi \left(𝒵\right)$
$𝒵$-block objective function $\psi$	$\psi$ splits as $h\left(Z_0\right)+g_1\left(Z_2\right)+g_2\left(Z_2\right)$
$h\left(Z_0\right)$	$h\left(Z_0\right)$ is proper, convex, and lower semicontinuous
$g_1\left(Z_S\right)$	$g_1$ is strongly convex, and either • $Z_S=Z_1$, or • $Z_S=(Z_0,Z_1)$
$g_2\left(Z_2\right)$	$g_2\left(Z_2\right)$ is Lipschitz differentiable
$Z_2$-operator $Q_2$	$Q_2$ is injective

Table 2. Table 2 . Contents of Assumption   2

	Assumption 2
$𝒳$-block function $f$	$f\left(𝒳\right)$ splits into $F(X_0,\dots ,X_n)+{\sum }_{i=0}^n{f}_i\left(X_i\right)$
$F(X_0,\dots ,X_n)$	$F(X_0,\dots ,X_n)$ is Lipschitz differentiable
$f_0\left(X_0\right),\dots ,f_n\left(X_n\right)$	$f_i\left(X_i\right)$ is proper and lower semicontinuous, and continuous on $\mathrm{dom}\left(f_i\right)$
$X_{\ell }$, for $1\le \ell \le n$	One of the following cases holds: Case 1: $F(X_0,\dots ,X_n)$ is independent of $X_{\ell }$, and $f_{\ell }\left(X_{\ell }\right)$ satisfies a strengthened convexity condition (see 5.13). Case 2: $f_{\ell }$ is either convex or Lipschitz differentiable. Viewing $A(𝒳,Z_0)+Q\left({𝒵}_{>}\right)=0$ as a system of constraints, there exists an constraint in the system, with index $r\left(\ell \right)$, such that in the $r\left(\ell \right)$-th constraint, we have $A_{r\left(\ell \right)}(𝒳,Z_0)=R_{\ell }\left(X_{\ell }\right)+A_{\ell }^{\prime }({𝒳}_{\ne \ell },Z_0)$ for an injective linear map $R_{\ell }$ and a multiaffine map $A_{\ell }^{\prime }$.
$Z_0$	One of the following cases holds: Case 1: $h\left(Z_0\right)$ satisfies a strengthened convexity condition (5.13). Case 2: $Z_0\in Z_S$, so $g_1\left(Z_S\right)$ is a strongly convex function of $Z_0$ and $Z_1$. Case 3: Viewing $A(𝒳,Z_0)+Q\left({𝒵}_{>}\right)=0$ as a system of constraints, there exists an index $r\left(0\right)$ such that $A_{r\left(0\right)}(𝒳,Z_0)=R_0\left(Z_0\right)+A_0^{\prime }\left(𝒳\right)$ for an injective linear map $R_0$ and multiaffine map $A_0^{\prime }$.

Equations

$$(P0)\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{x,y}&f(x)+g(y)\\ &Ax+By-b=0.\end{array}\right.$$

$$\mathcal{L}(x,y,w)=f(x)+g(y)+⟨w,Ax+By-b⟩+\frac{\rho }{2}\Vert Ax+By-b{\Vert }^2.$$

$$(P1)\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{x_{1},x_{2},\ldots,x_{n}}&f_{1}(x_{1})+f_{2}(x_{2})+\ldots+f_{n}(x_{n})\\ &A_{1}x_{1}+A_{2}x_{2}+\ldots+A_{n}x_{n}-b=0\end{array}\right.$$

$$(P2)\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{x_{1},x_{2},\ldots,x_{n}}&f(x_{1},\ldots,x_{n})\\ &C(x_{1},\ldots,x_{n})=0.\end{array}\right.$$

$$\mathcal{L}(x_1,\dots ,x_n,w)=f(x_1,\dots ,x_n)+⟨w,C(x_1,\dots ,x_n)⟩+\frac{\rho }{2}\Vert C(x_1,\dots ,x_n){\Vert }^2,$$

$$(\text{NMF1})\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{Z,X,Y}&\frac{1}{2}\|Z-B\|^{2}\\ &Z=XY,X\geq 0,Y\geq 0,\end{array}\right.$$

$$X_i\mapsto \mathcal{M}({\overline{X}}_1,\dots ,{\overline{X}}_{i-1},X_i,{\overline{X}}_{i+1},\dots ,{\overline{X}}_n)$$

$$(P)\hskip 15.00002pt\left\{\begin{array}[]{rll}\displaystyle\inf_{\mathcal{X},\mathcal{Z}}&\phi(\mathcal{X},\mathcal{Z})\\ &A(\mathcal{X},Z_{0})+Q(\mathcal{Z}_{>})=0,\end{array}\right.$$

$$\varphi (\mathcal{X},\mathcal{Z})$$

$$\text{and}A(\mathcal{X},Z_0)+Q({\mathcal{Z}}_{>})$$

$$\mathcal{L}(\mathcal{X},\mathcal{Z},\mathcal{W})=\varphi (\mathcal{X},\mathcal{Z})+⟨\mathcal{W},A(\mathcal{X},Z_0)+Q({\mathcal{Z}}_{>})⟩+\frac{\rho }{2}\Vert A(\mathcal{X},Z_0)+Q({\mathcal{Z}}_{>}){\Vert }^2,$$

$$\displaystyle(\text{DL})\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{X,Y}&\iota_{S}(X)+\|Y\|_{1}+\frac{\mu}{2}\|XY-B\|^{2},\\ \end{array}\right.$$

$$(\text{DL1})\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{Z,X,Y}&\iota_{S}(X)+\|Y\|_{1}+\frac{1}{2}\|Z-B\|^{2}\\ &Z=XY.\end{array}\right.$$

$$(\text{RPCA1})\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{L,S}&\|L\|_{*}+\lambda\|S\|_{1}\\ &L+S=B\end{array}\right.$$

$$(\text{RPCA2})\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{U,V,S}&\frac{1}{2}(\|U\|_{F}^{2}+\|V\|_{F}^{2})+\lambda\|S\|_{1}\\ &UV^{T}+S=B\\ &U\in\mathbb{R}^{m\times k},V\in\mathbb{R}^{n\times n},S\in\mathbb{R}^{m\times n}\end{array}\right.$$

$$(\text{MC1})\hskip 15.00002pt\left\{\begin{array}[]{rll}\sup\limits_{Z,x,y,s}&\frac{1}{2}\sum\limits_{uv\in E}w_{uv}(1-Z_{uv})+\frac{\mu_{1}}{2}\sum\limits_{u\in V}(Z_{uu}-1)^{2}+\frac{\mu_{2}}{2}\|s\|^{2}\\ &Z=xy^{T},\ x-y=s.\end{array}\right.$$

$$(\text{RP})\hskip 15.00002pt\left\{\begin{array}[]{l}x_{i}(\Sigma x)_{i}=x_{j}(\Sigma x)_{j}\hskip 15.00002pt\forall i,j\\ a\leq x\leq b,\hskip 15.00002ptx_{1}+\ldots+x_{n}=1\end{array}\right.$$

$$(\text{RP}1)\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{x,y,z,s}&\iota_{X}(x)+\frac{\mu}{2}(\|z\|^{2}+\|s\|^{2})\\ &P(x\circ y)=z\\ &y-\Sigma x=s\end{array}\right.$$

$$(\text{DNN}1)\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{\{X_{\ell}\},\{a_{\ell}\},\{z_{\ell}\}}&E(z_{L},y)+R(X_{1},\ldots,X_{L})\\ &z_{\ell}-X_{\ell}a_{\ell-1}=0\text{ for }1\leq\ell\leq L\\ &a_{\ell}-h(z_{\ell})=0\text{ for }1\leq\ell\leq L-1.\end{array}\right.$$

$$(\text{DNN}2)\hskip 15.00002pt\left\{\begin{array}[]{rll}\inf\limits_{\{X_{\ell}\},\{a_{\ell}\},\{z_{\ell}\}}&E(z_{L},y)+R(X_{1},\ldots,X_{L})+\frac{\mu}{2}\sum\limits_{\ell=1}^{L-1}(a_{\ell}-h(z_{\ell}))^{2}\\ &z_{\ell}-X_{\ell}a_{\ell-1}=0\text{ for }1\leq\ell\leq L.\end{array}\right.$$

$$\psi (\mathcal{Z})=h(Z_0)+g_1(Z_S)+g_2(Z_2)$$

$$0\in {\partial }_{\mathcal{Z}}\psi ({\mathcal{Z}}^{\ast })+C_{{\mathcal{X}}^{\ast }}^T{\mathcal{W}}^{\ast }$$

$$\underset{\mathcal{Z}}{\min }\{\psi (\mathcal{Z}):A({\mathcal{X}}^{\ast },Z_0)+Q({\mathcal{Z}}_{>})=0\}.$$

$$\frac{\sigma \rho }{2}-\frac{M_2^2}{\sigma \rho }>\frac{M_2}{2}\text{ and }\rho >\max \left\{\frac{2M_1{\kappa }_1}{{\lambda }_{++}(Q_1^T{Q}_1)},\frac{1}{2}(M_1+M_2)\max \left\{{\sigma }^{-1},\frac{(1+2{\kappa }_1{)}^2}{{\lambda }_{++}(Q_1^T{Q}_1)}\right\}\right\}.$$

$$f(\mathcal{X})=F(X_0,\dots ,X_n)+\sum _{i=0}^n{f}_i(X_i)$$

$$A_{r(\ell )}(\mathcal{X},Z_0)=R_{\ell }(X_{\ell })+A_{\ell }^{\prime }({\mathcal{X}}_{\ne \ell },Z_0)$$

$$0\in {\nabla }_{X_i}F({\mathcal{X}}^{\ast })+{\partial }_{X_i}{f}_i(X_i^{\ast })+A_{X_i,({\mathcal{X}}_{\ne i}^{\ast },Z_0^{\ast })}^T{\mathcal{W}}^{\ast }$$

$$\underset{X_i}{\min }\{F({\mathcal{X}}_{\ne i}^{\ast },X_i)+f_i(X_i):A({\mathcal{X}}_{\ne i}^{\ast },X_i,Z_0^{\ast })+Q({\mathcal{Z}}_{>}^{\ast })=0\}.$$

$$\mu_{\ell}=\left\{\begin{array}[]{cl}0&\text{if $f_{\ell}$ convex }\\ M_{\ell}&\text{if $f_{\ell}$ nonconvex, Lipschitz differentiable. }\end{array}\right.$$

$$\underset{x,y}{\min }\{x^2+y^2:xy=1\}.$$

$$\mathcal{L}({\mathcal{X}}_{<\ell }^{+},X_{\ell },{\mathcal{X}}_{>\ell },\mathcal{Z},\mathcal{W})-\mathcal{L}({\mathcal{X}}_{<\ell }^{+},X_{\ell }^{+},{\mathcal{X}}_{>\ell },\mathcal{Z},\mathcal{W})\ge \Delta (\Vert X_{\ell }-X_{\ell }^{+}\Vert )$$

Full text

† Department of Industrial Engineering and Operations Research, Columbia University. Research of this author was supported in part by NSF Grant CCF-1527809.

‡ Department of Industrial and Systems Engineering, Lehigh University. Research of this author was supported in part by DOE Grant DE-SC0010615 and NSF Grant CCF-1618717.

1. Introduction

The alternating direction method of multipliers (ADMM) is an iterative method which, in its original form, solves linearly-constrained separable optimization problems with the following structure:

[TABLE]

The augmented Lagrangian $\mathcal{L}$ of the problem $(P0)$, for some penalty parameter $\rho>0$, is defined to be

[TABLE]

In iteration $k$, with the iterate $({x}^{k},{y}^{k},{w}^{k})$, ADMM takes the following steps:

1. (1)

Minimize $\mathcal{L}(x,{y}^{k},{w}^{k})$ with respect to $x$ to obtain ${x}^{k+1}$. 2. (2)

Minimize $\mathcal{L}({x}^{k+1},y,{w}^{k})$ with respect to $y$ to obtain ${y}^{k+1}$. 3. (3)

Set ${w}^{k+1}\leftarrow{w}^{k}+\rho(A{x}^{k+1}+B{y}^{k+1}-b)$.

ADMM was first proposed [20, 21] for solving variational problems, and was subsequently applied to convex optimization problems with two blocks as in $(P0)$. Several techniques can be used to analyze this case, including an operator-splitting approach [16, 41]. The survey articles [17, 6] provide convergence proofs from several viewpoints, and discuss numerous applications of ADMM. More recently, there has been considerable interest in extending ADMM convergence guarantees when solving problems with multiple blocks and nonconvex objective functions. ADMM directly extends to the problem

[TABLE]

by minimizing $\mathcal{L}(x_{1},\ldots,x_{n},w)$ with respect to $x_{1},x_{2},\ldots,x_{n}$ successively. The multiblock problem turns out to be significantly different from the classical 2-block problem, even when the objective function is convex; for example, [9] exhibits an example with $n=3$ blocks and $f_{1},f_{2},f_{3}\equiv 0$ for which ADMM diverges for any value of $\rho$. Under certain conditions, the unmodified 3-block ADMM does converge. In [39], it is shown that if $f_{3}$ is strongly convex with condition number $\kappa\in[1,1.0798)$ (among other assumptions), then 3-block ADMM is globally convergent. If $f_{1},\ldots,f_{n}$ are all strongly convex, and $\rho>0$ is sufficiently small, then [24] shows that multiblock ADMM is convergent. Other works along these lines include [37, 38, 34].

In the absence of strong convexity, modified versions of ADMM have been proposed that can accommodate multiple blocks. In [13] a new type of 3-operator splitting is introduced that yields a convergent 3-block ADMM (see also [46] for a proof that a ‘lifting-free’ 3-operator extension of Douglas-Rachford splitting does not exist). Convergence guarantees for multiblock ADMM can also be achieved through variants such as proximal ADMM, majorized ADMM, linearized ADMM [49, 36, 14, 10, 40, 5], and proximal Jacobi ADMM [14, 55, 52].

ADMM has also been extended to problems with nonconvex objective functions. In [25], it is proved that ADMM converges when the problem $(P1)$ is either a nonconvex consensus or sharing problem, and [57] proves convergence under more general conditions on $f_{1},\ldots,f_{n}$ and $A_{1},\ldots,A_{n}$. Proximal ADMM schemes for nonconvex, nonsmooth problems are considered in [33, 60, 28, 5]. More references on nonconvex ADMM, and comparisons of the assumptions used, can be found in [57].

In all of the work mentioned above, the system of constraints $C(x_{1},\ldots,x_{n})=0$ is assumed to be linear. Consequently, when all variables other than $x_{i}$ have fixed values, $C(x_{1},\ldots,x_{n})$ becomes an affine function of $x_{i}$. However, this holds for more general constraints $C(\cdot)$ in the much larger class of multiaffine maps (see Section 2). Thus, it seems reasonable to expect that ADMM would behave similarly when the constraints $C(x_{1},\ldots,x_{n})=0$ are permitted to be multiaffine. To be precise, consider a more general problem than $(P1)$ of the form

[TABLE]

The augmented Lagrangian for $(P2)$ is

[TABLE]

and ADMM for solving this problem is specified in Algorithm 1.

While many problems can be modeled with multiaffine constraints, existing work on ADMM for solving multiaffine constrained problems appears to be limited. Boyd et al. [6] propose solving the nonnegative matrix factorization problem formulated as a problem with biaffine constraints, i.e.,

[TABLE]

by applying ADMM with alternating minimization on the blocks $Y$ and $(X,Z)$. The convergence of ADMM employed to solve the (NMF1) problem appears to have been an open question until a proof was given in [23]111[23] shows that every limit point of ADMM for the problem (NMF) is a constrained stationary point, but does not show that such limit points necessarily exist.. A method derived from ADMM has also been proposed for optimizing a biaffine model for training deep neural networks [53]. For general nonlinear constraints, a framework for “monitored” Lagrangian-based multiplier methods was studied in [4].

In this paper, we establish the convergence of ADMM for a broad class of problems with multiaffine constraints. Our assumptions are similar to those used in [57] for nonconvex ADMM; in particular, we do not make any assumption about the iterates generated by the algorithm. Hence, these results extend the applicability of ADMM to a larger class of problems which naturally have multiaffine constraints. Moreover, we prove several results about ADMM in Section 6 that hold in even more generality, and thus may be useful for analyzing ADMM beyond the setting considered here.

1.1. Organization of this paper

In Section 2, we define multilinear and multiaffine maps, and specify the precise structure of the problems that we consider. In Section 3, we provide several examples of problems that can be formulated with multiaffine constraints. In Section 4, we state our assumptions and main results (i.e., Theorems 4.1, 4.3 and 4.5). In Section 5, we present a collection of necessary technical material. In Section 6, we prove several results about ADMM that hold under weak conditions on the objective function and constraints. Finally, in Section 7, we complete the proof of our main convergence theorems (Theorems 4.1, 4.3 and 4.5), by applying the general techniques developed in Section 6. Appendix A contains proofs of technical lemmas. Appendix B presents an alternative biaffine formulation for deep neural network training. Appendix C organizes the major assumptions in tabular form. Appendix D presents additional formulations of problems where all ADMM subproblems have closed-form solutions. Appendix E presents several numerical experiments.

1.2. Notation and Definitions

We consider only finite-dimensional real vector spaces. The symbols $\mathbb{E},\mathbb{E}_{1},\ldots,\mathbb{E}_{n}$ denote finite-dimensional Hilbert spaces, equipped with inner products $\langle\cdot,\cdot\rangle$. By default, we use the standard inner product on $\mathbb{R}^{n}$ and the trace inner product $\langle X,Y\rangle=\operatorname{Tr}(Y^{T}X)$ on the matrix space. Unless otherwise specified, the norm $\|\cdot\|$ is always the induced norm of the inner product. When $A$ is a matrix or linear map, $\|A\|_{op}$ denotes the $L_{2}$ operator norm, and $\|A\|_{*}$ denotes the nuclear norm (the sum of the singular values of $A$). Fixed bases are assumed, so we freely use various properties of a linear map $A$ that depend on its representation (such as $\|A\|_{op}$), and view $A$ as a matrix as required.

For $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\infty\}$, the effective domain $\operatorname{dom}(f)$ is the set $\{x:f(x)<\infty\}$. The image of a function $f$ is denoted by $\operatorname{Im}(f)$. Similarly, when $A$ is a linear map represented by a matrix, $\operatorname{Im}(A)$ is the column space of $A$. We use $\operatorname{Null}(A)$ to denote the null space of $A$. The orthogonal complement of a linear subspace $U$ is denoted $U^{\perp}$.

To distinguish the derivatives of smooth (i.e., continuously differentiable) functions from subgradients, we use the notation $\nabla_{X}$ for partial differentiation with respect to $X$, and reserve the symbol $\partial$ for the set of general subgradients (Section 5.1); hence, the use of $\nabla f$ serves as a reminder that $f$ is assumed to be smooth. A function $f$ is Lipschitz differentiable if it is differentiable and its gradient is Lipschitz continuous.

When $\mathcal{X}$ is a tuple of variables $\mathcal{X}=(X_{0},\ldots,X_{n})$, we write $\mathcal{X}_{\neq\ell}$ for $(X_{i}:i\neq\ell)$. Similarly, $\mathcal{X}_{>\ell}$ and $\mathcal{X}_{<\ell}$ represent $(X_{i}:i>\ell)$ and $(X_{i}:i<\ell)$ respectively.

We use the term constrained stationary point for a point satisfying necessary first-order optimality conditions; this is a generalization of the Karush-Kuhn-Tucker (KKT) necessary conditions to nonsmooth problems. For the problem $\min_{x}\{f(x):C(x)=0\}$, where $C$ is smooth and $f$ possesses general subgradients, $x^{\ast}$ is a constrained stationary point if $C(x^{\ast})=0$ and there exists $w^{\ast}$ with $0\in\partial f(x^{\ast})+\nabla C(x^{\ast})^{T}w^{\ast}$.

2. Multiaffine Constrained Problems

The central objects of this paper are multilinear and multiaffine maps, which generalize linear and affine maps.

Definition 2.1**.**

A map $\mathcal{M}:\mathbb{E}_{1}\oplus\ldots\oplus\mathbb{E}_{n}\rightarrow\mathbb{E}$ is multilinear if, for all $i\leq n$ and all points $(\overline{X}_{1},\ldots,\overline{X}_{i-1},\overline{X}_{i+1},\ldots,\overline{X}_{n})\in\bigoplus_{j\neq i}E_{j}$, the map $\mathcal{M}_{i}:\mathbb{E}_{i}\rightarrow\mathbb{E}$ given by

[TABLE]

is linear. Similarly, $\mathcal{M}$ is multiaffine if the map $\mathcal{M}_{i}$ is affine for all $i$ and all points of $\bigoplus_{j\neq i}\mathbb{E}_{j}$. In particular, when $n=2$, we say that $\mathcal{M}$ is bilinear/biaffine.

We consider the convergence of ADMM for problems of the form:

[TABLE]

where $\mathcal{X}=(X_{0},\ldots,X_{n})$, $\mathcal{Z}=(Z_{0},\mathcal{Z}_{>})$, $\mathcal{Z}_{>}=(Z_{1},Z_{2})$,

[TABLE]

with $A_{1}$ and $A_{2}$ being multiaffine maps and $Q_{1}$ and $Q_{2}$ being linear maps. The augmented Lagrangian $\mathcal{L}(\mathcal{X},\mathcal{Z},\mathcal{W})$, with penalty parameter $\rho>0$, is given by

[TABLE]

where $\mathcal{W}=(W_{1},W_{2})$ are Lagrange multipliers.

We prove that Algorithm 1 converges to a constrained stationary point under certain assumptions on $\phi,A$, and $Q$, which are described in Section 4. Moreover, since the constraints are nonlinear, there is a question of constraint qualifications, which we address in Lemma 5.4.

We adopt the following notation in the context of ADMM. The variables in the $k$-th iteration are denoted ${\mathcal{X}}^{k},{\mathcal{Z}}^{k},{\mathcal{W}}^{k}$ (with ${X}^{k}_{i},{Z}^{k}_{i},{W}^{k}_{i}$ for the $i$-th variable in each component). When analyzing a single iteration, the index $k$ is omitted, and we write $X={X}^{k}$ and $X^{+}={X}^{k+1}$. Similarly, we write ${\mathcal{L}}^{k}=\mathcal{L}({\mathcal{X}}^{k},{\mathcal{Z}}^{k},{\mathcal{W}}^{k})$ and will refer to $\mathcal{L}={\mathcal{L}}^{k}$ and $\mathcal{L}^{+}={\mathcal{L}}^{k+1}$ for values within a single iteration.

3. Examples of Applications

In this section, we describe several problems with multiaffine constraints, and show how they can be formulated and solved by ADMM. Many important applications of ADMM involve introducing auxiliary variables so that all subproblems have closed-form solutions; we describe several such reformulations in appendix D that have this property.

3.1. Representation Learning

Given a matrix $B$ of data, it is often desirable to represent $B$ in the form $B=X\ast Y$, where $\ast$ is a bilinear map and the matrices $X,Y$ have some desirable properties. Two important applications follow:

1. (1)

Nonnegative matrix factorization (NMF) [31, 32] expresses $B$ as a product of nonnegative matrices $X\geq 0,Y\geq 0$. 2. (2)

Inexact dictionary learning (DL) [43] expresses every element of $B$ as a sparse combination of atoms from a dictionary $X$. It is typically formulated as

[TABLE]

where $\iota_{S}$ is the indicator function for the set $S$ of matrices whose columns have unit $L_{2}$ norm, and here $\|Y\|_{1}$ is the entrywise 1-norm $\sum_{i,j}|Y_{ij}|$. The parameter $\mu$ is an input that sets the balance between trying to recover $B$ with high fidelity versus finding $Y$ with high sparsity.

Problems of this type can be modeled with bilinear constraints. As already mentioned in Section 1, [6, 23] propose the bilinear formulation (NMF1) for nonnegative matrix factorization. The inexact dictionary learning problem can similarly be formulated as:

[TABLE]

Other variants of dictionary learning such as convolutional dictionary learning (CDL), that cannot readily be handled by the method in [43], have a biaffine formulation which is nearly identical to (DL1), and can be solved using ADMM. For more information on dictionary learning, see [18, 50, 51, 43, 56].

3.2. Non-Convex Reformulations of Convex Problems

Recently, various low-rank matrix and tensor recovery problems have been shown to be efficiently solvable by applying first-order methods to nonconvex reformulations of them. For example, the convex Robust Principal Component Analysis (RPCA) [26, 8] problem

[TABLE]

can be reformulated as the biaffine problem

[TABLE]

as long as $k\geq\operatorname{rank}(L^{\ast})$, where $L^{\ast}$ is an optimal solution of (RPCA1). See [15] for a proof of this, and applications of the factorization $UV^{T}$ to other problems. This is also related to the Burer-Monteiro approach [7] for semidefinite programming. We remark that (RPCA2) does not satisfy all the assumptions needed for the convergence of ADMM (see 1.3 and Section 4.2), so slack variables must be added.

3.3. Max-Cut

Given a graph $G=(V,E)$ and edge weights $w\in\mathbb{R}^{E}$, the (weighted) maximum cut problem is to find a subset $U\subseteq V$ so that $\sum\limits_{u\in U,v\notin U}w_{uv}$ is maximized. This problem is well-known to be NP-hard [30]. An approximation algorithm using semidefinite programming can be shown to achieve an approximation ratio of roughly $0.878$ [22]. Applying the Burer-Monteiro approach [7] to the max-cut semidefinite program [22] with a rank-one constraint, and introducing a slack variable (see 1.2), we obtain the problem

[TABLE]

It is easy to verify that all subproblems have very simple closed-form solutions.

3.4. Risk Parity Portfolio Selection

Given assets indexed by $\{1,\ldots,n\}$, the goal of risk parity portfolio selection is to construct a portfolio weighting $x\in\mathbb{R}^{n}$ in which every asset contributes an equal amount of risk. This can be formulated with quadratic constraints; see [3] for details. The feasibility problem in [3] is

[TABLE]

where $\Sigma$ is the (positive semidefinite) covariance matrix of the asset returns, and $a$ and $b$ contain lower and upper bounds on the weights, respectively. The authors in [3] introduce a variable $y=x$ and solve (RP) using ADMM by replacing the quadratic risk-parity constraint by a fourth-order penalty function $f(x,y,\theta)=\sum_{i=1}^{n}(x_{i}(\Sigma y)_{i}-\theta)^{2}$. To rewrite this problem with a bilinear constraint, let $\circ$ denote the Hadamard product $(x\circ y)_{i}=x_{i}y_{i}$ and let $P$ be the matrix $\begin{pmatrix}0&0\\ e_{n-1}&-I_{n-1}\end{pmatrix}$, where $e_{n}$ is the all-ones vector of length $n$. Let $X$ be the set of permissible portfolio weights $X=\{x\in\mathbb{R}^{n}:a\leq x\leq b\}\cap\{x\in\mathbb{R}^{n}:e_{n}^{T}x=1\}$, and let $\iota_{X}$ be its indicator function. Then we obtain the problem

[TABLE]

where we have introduced a slack variable $s$ (see 1.2).

3.5. Training Neural Networks

An alternating minimization approach is proposed in [53] for training deep neural networks. By decoupling the linear and nonlinear elements of the network, the backpropagation required to compute the gradient of the network is replaced by a series of subproblems which are easy to solve and readily parallelized. For a network with $L$ layers, let $X_{\ell}$ be the matrix of edge weights for $1\leq\ell\leq L$, and let $a_{\ell}$ be the output of the $\ell$-th layer for $0\leq\ell\leq L-1$. Deep neural networks are defined by the structure $a_{\ell}=h(X_{\ell}a_{\ell-1})$, where $h(\cdot)$ is an activation function, which is often taken to be the rectified linear unit (ReLU) $h(z)=\max\{z,0\}$. The splitting used in [53] introduces new variables $z_{\ell}$ for $1\leq\ell\leq L$ so that the network layers are no longer directly connected, but are instead coupled through the relations $z_{\ell}=X_{\ell}a_{\ell-1}$ and $a_{\ell}=h(z_{\ell})$.

Let $E(\cdot,\cdot)$ be an error function, and $R$ a regularization function on the weights. Given a matrix of labeled training data $(a_{0},y)$, the learning problem is

[TABLE]

The algorithm proposed in [53] does not include any regularization $R(\cdot)$, and replaces both sets of constraints by quadratic penalty terms in the objective, while maintaining Lagrange multipliers only for the final constraint $z_{L}=W_{L}a_{L-1}$. However, since all of the equations $z_{\ell}=X_{\ell}a_{\ell-1}$ are biaffine, we can include them in a biaffine formulation of the problem:

[TABLE]

To adhere to our convergence theory, it would be necessary to apply smoothing (such as Nesterov’s technique [44]) when $h(z)$ is nonsmooth, as is the ReLU. Alternatively, the ReLU can be replaced by an approximation using nonnegativity constraints (see Appendix B). In practice [53, §7], using the ReLU directly yields simple closed-form solutions, and appears to perform well experimentally. However, no proof of the convergence of the algorithm in [53] is provided.

4. Main Results

In this section, we state our assumptions and main results. We will show that ADMM (Algorithm 2) applied to solve a multiaffine constrained problem of the form $(P)$ (refer to page 2) produces a bounded sequence $\{({X}^{k},{\mathcal{Z}}^{k})\}_{k=0}^{\infty}$, and that every limit point $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast})$ is a constrained stationary point. While there are fairly general conditions under which $\mathcal{Z}^{\ast}$ satisfies first-order optimality conditions (see Assumption 1 and the corresponding discussion in Section 4.2 of tightness), the situation with $\mathcal{X}^{\ast}$ is more complicated because of the many possible structures of multiaffine maps. Accordingly, we divide the convergence proof into two results. Under one broad set of assumptions, we prove that limit points exist, are feasible, and that $\mathcal{Z}^{\ast}$ is a blockwise constrained stationary point for the problem with $\mathcal{X}$ fixed at $\mathcal{X}^{\ast}$ (Theorem 4.1). Then, we present a set of easily-verifiable conditions under which $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast})$ is also a constrained stationary point (Theorem 4.3). If the augmented Lagrangian has additional geometric properties (namely, the Kurdyka-Łojasiewicz property (Section 5.5)), then $\{({X}^{k},{\mathcal{Z}}^{k})\}_{k=0}^{\infty}$ converges to a single limit point $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast})$ (Theorem 4.5).

4.1. Assumptions and Main Results

We consider two sets of assumption for our analysis. We provide intuition and further discussion of them in Section 4.2. (See Section 5 for definitions related to convexity and differentiability.)

Assumption 1**.**

Solving problem $(P)$ (refer to page 2), the following hold.

A 1.1**.**

For sufficiently large $\rho$, every ADMM subproblem attains its optimal value.

A 1.2**.**

$\operatorname{Im}(Q)\supseteq\operatorname{Im}(A)$.

A 1.3**.**

The following statements regarding the objective function $\phi$ and $Q_{2}$ hold:

1. (1)

$\phi$* is coercive on the feasible region $\Omega=\{(\mathcal{X},\mathcal{Z}):A(\mathcal{X},Z_{0})+Q(\mathcal{Z}_{>})=0\}$.* 2. (2)

$\psi(\mathcal{Z})$* can be written in the form*

[TABLE]

where

1. (a)

$h$* is proper, convex, and lower semicontinuous.* 2. (b)

$Z_{S}$* represents either $Z_{1}$ or $(Z_{0},Z_{1})$ and $g_{1}$ is $(m_{1},M_{1})$-strongly convex. That is, either $g_{1}(Z_{1})$ is a strongly convex function of $Z_{1}$ or $g_{1}(Z_{0},Z_{1})$ is a strongly convex function of $(Z_{0},Z_{1})$.* 3. (c)

$g_{2}$* is $M_{2}$-Lipschitz differentiable.* 3. (3)

$Q_{2}$* is injective.*

While Assumption 1 may appear to be complicated, it is no stronger than the conditions used in analyzing nonconvex, linearly-constrained ADMM. A detailed comparison is given in Section 4.2.

Under Assumption 1, Algorithm 2 produces a sequence which has limit points, and every limit point $({\mathcal{X}}^{\ast},{\mathcal{Z}}^{\ast})$ is feasible with ${\mathcal{Z}}^{\ast}$ a constrained stationary point for problem $(P)$ with $\mathcal{X}$ fixed to $\mathcal{X}^{\ast}$.

Theorem 4.1**.**

Suppose that Assumption 1 holds. For sufficiently large $\rho$, the sequence $\{({\mathcal{X}}^{k},{\mathcal{Z}}^{k},{\mathcal{W}}^{k})\}_{k=0}^{\infty}$ produced by ADMM is bounded, and therefore has limit points. Every limit point $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast},\mathcal{W}^{\ast})$ satisfies $A(\mathcal{X}^{\ast},Z_{0}^{\ast})+Q(\mathcal{Z}_{>}^{\ast})=0$. There exists a sequence ${v}^{k}\in\partial_{\mathcal{Z}}\mathcal{L}({\mathcal{X}}^{k},{\mathcal{Z}}^{k},{\mathcal{W}}^{k})$ such that ${v}^{k}\rightarrow 0$, and thus

[TABLE]

where $C_{\mathcal{X}^{\ast}}$ is the linear map $\mathcal{Z}\mapsto A(\mathcal{X}^{\ast},Z_{0})+Q(\mathcal{Z}_{>})$ and $C_{\mathcal{X}^{\ast}}^{T}$ is its adjoint. That is, $\mathcal{Z}^{\ast}$ is a constrained stationary point for the problem

[TABLE]

Remark 4.2**.**

Let $\sigma:=\lambda_{min}(Q_{2}^{T}Q_{2})$ 222See Section 5.4 for the definition of $\lambda_{min}$ and $\lambda_{++}$. and $\kappa_{1}:=\frac{M_{1}}{m_{1}}$. One can check that it suffices to choose $\rho$ so that

[TABLE]

Note that Assumption 1 makes very few assumptions about $f(\mathcal{X})$ and the map $A$ as a function of $\mathcal{X}$, other than that $A$ is multiaffine. In Section 6, we develop general techniques for proving that $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast})$ is a constrained stationary point. We now present an easily checkable set of conditions, that ensure that the requirements for those techniques are satisfied.

Assumption 2**.**

Solving problem $(P)$, Assumption 1 and the following hold.

A 2.1**.**

The function $f(\mathcal{X})$ splits into

[TABLE]

where $F$ is $M_{F}$-Lipschitz differentiable, the functions $f_{0}$, $f_{1}$, $\ldots$, and $f_{n}$ are proper and lower semicontinuous, and each $f_{i}$ is continuous on $\operatorname{dom}(f_{i})$.

A 2.2**.**

For each $1\leq\ell\leq n$,333Note that we have deliberately excluded $\ell=0$. 2.2 is not required to hold for $X_{0}$. at least one of the following two conditions444That is, either (1a) and (1b) hold, or (2a) and (2b) hold. holds:

1. (1)

2. (a)

$F(X_{0},\ldots,X_{n})$* is independent of $X_{\ell}$.* 2. (b)

$f_{\ell}(X_{\ell})$* satisfies a strengthened convexity condition (5.13).* 2. (2)

1. (a)

*Viewing $A(\mathcal{X},Z_{0})+Q(\mathcal{Z}_{>})=0$ as a system of constraints 555As an illustrative example, a problem may be formulated with constraints $X_{0}X_{1}+Z_{1}=0,X_{0}+P_{1}(X_{1})+Z_{2}=0,X_{0}X_{2}+Z_{3}=0,P_{2}(X_{2})+Z_{4}=0$, where $P_{1},P_{2}$ are injective linear maps. The notation $A(\mathcal{X},Z_{0})+Q(\mathcal{Z}_{>})$ denotes the concatenation of these equations, which can also be seen naturally as a system of four constraints. In this case, the indices $r(\ell)\in\{1,2,3,4\}$, and 2.2(2a) is satisfied by the second constraint $X_{0}+P_{1}(X_{1})+Z_{2}=0$ for the variables $X_{0},X_{1}$ (i.e. $r(0)=r(1)=2$ and $R_{0}=I,R_{1}=P_{1}$), and by the fourth constraint $P_{2}(X_{2})+Z_{4}=0$ for $X_{2}$. * , there exists an index $r(\ell)$ such that in the $r(\ell)$-th constraint,

[TABLE]

for an injective linear map $R_{\ell}$ and a multiaffine map $A^{\prime}_{\ell}$. In other words, the only term in $A_{r(\ell)}$ that involves $X_{\ell}$ is an injective linear map $R_{\ell}(X_{\ell})$. 2. (b)

$f_{\ell}$* is either convex or $M_{\ell}$-Lipschitz differentiable.*

A 2.3**.**

At least one of the following holds for $Z_{0}$:

1. (1)

$h(Z_{0})$* satisfies a strengthened convexity condition (5.13).* 2. (2)

$Z_{0}\in Z_{S}$, so $g_{1}(Z_{S})$ is a strongly convex function of $Z_{0}$ and $Z_{1}$. 3. (3)

Viewing $A(\mathcal{X},Z_{0})+Q(\mathcal{Z}_{>})=0$ as a system of constraints, there exists an index $r(0)$ such that $A_{r(0)}(\mathcal{X},Z_{0})=R_{0}(Z_{0})+A^{\prime}_{0}(\mathcal{X})$ for an injective linear map $R_{0}$ and multiaffine map $A_{0}^{\prime}$.

With these additional assumptions on $f$ and $A$, we have that every limit point $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast})$ is a constrained stationary point of problem $(P)$.

Theorem 4.3**.**

Suppose that Assumption 2 holds (and hence, Assumption 1 and Theorem 4.1). Then for sufficiently large $\rho$, there exists a sequence ${v}^{k}\in\partial\mathcal{L}({\mathcal{X}}^{k},{\mathcal{Z}}^{k},{\mathcal{W}}^{k})$ with ${v}^{k}\rightarrow 0$, and thus every limit point $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast})$ is a constrained stationary point of problem $(P)$. Thus, in addition to (4.1), $\mathcal{X}^{\ast}$ satisfies, for each $0\leq i\leq n$,

[TABLE]

where $A_{X_{i},(\mathcal{X}^{\ast}_{\neq i},Z_{0}^{\ast})}$ is the $X_{i}$-linear term of $\mathcal{X}\mapsto A(\mathcal{X},Z_{0})$ evaluated at $(\mathcal{X}_{\neq i}^{\ast},Z_{0}^{\ast})$ (see 5.6) and $A_{X_{i},(\mathcal{X}^{\ast}_{\neq i},Z_{0}^{\ast})}^{T}$ is its adjoint. That is, for each $0\leq i\leq n$, $X^{\ast}_{i}$ is a constrained stationary point for the problem

[TABLE]

Remark 4.4**.**

One can check that it suffices to choose $\rho$ so that, in addition to (4.2), we have $\rho>\max\{\lambda_{min}^{-1}(R_{\ell}^{T}R_{\ell})(\mu_{\ell}+M_{F})\}$, where the maximum is taken over all $\ell$ for which 2.2(2) holds, and

[TABLE]

It is well-known that when the augmented Lagrangian has a geometric property known as the Kurdyka-Łojasiewicz (K-Ł) property (see Section 5.5), which is the case for many optimization problems that occur in practice, then results such as Theorem 4.3 can typically be strengthened because the limit point is unique.

Theorem 4.5**.**

Suppose that $\mathcal{L}(\mathcal{X},\mathcal{Z},\mathcal{W})$ is a K-Ł function. Suppose that Assumption 2 holds, and furthermore, that 2.2(2) holds for all $X_{0},X_{1},\ldots,X_{n}$666Note that $X_{0}$ is included here, unlike in Assumption 2., and 2.3(2) holds. Then for sufficiently large $\rho$, the sequence $\{({\mathcal{X}}^{k},{\mathcal{Z}}^{k},{\mathcal{W}}^{k})\}_{k=0}^{\infty}$ produced by ADMM converges to a unique constrained stationary point $(\mathcal{X}^{\ast},\mathcal{Z}^{\ast},\mathcal{W}^{\ast})$.

In Section 6, we develop general properties of ADMM that hold without relying on Assumption 1 or Assumption 2. In Section 7, the general results are combined with Assumption 1 and then with Assumption 2 to prove Theorem 4.1 and Theorem 4.3, respectively. Finally, we prove Theorem 4.5 assuming that the augmented Lagrangian is a K-Ł function. The results of Section 6 may also be useful for analyzing ADMM, since the assumptions required are weak.

4.2. Discussion of Assumptions

Assumptions 1 and 2 are admittedly long and somewhat involved. In this section, we will discuss them in detail and explore the extent to which they are tight. Again, we wish to emphasize that despite the additional complexity of multiaffine constraints, the basic content of these assumptions is fundamentally the same as in the linear case. There is also a relation between Assumption 2 and proximal ADMM, by which 2.2(2) can be viewed as introducing a proximal term. This is described in Section 4.2.6.

4.2.1. Assumption 1.1

This assumption is necessary for ADMM (Algorithm 2) to be well-defined. We note that this can fail in surprising ways; for instance, the conditions used in [6] are insufficient to guarantee that the ADMM subproblems have solutions. In [11], an example is constructed which satisfies the conditions in [6], and yet the ADMM subproblem fails to attain its (finite) optimal value.

4.2.2. Assumption 1.2

The condition that $\operatorname{Im}(Q)\supseteq\operatorname{Im}(A)$ plays a crucial role at multiple points in our analysis because $\mathcal{Z}_{>}$, a subset of the final block of variables, has a close relation to the dual variables $\mathcal{W}$. It would greatly broaden the scope of ADMM, and simplify modeling, if this condition could be relaxed, but unfortunately this condition is tight for general problems. The following example demonstrates that ADMM is not globally convergent when 1.2 does not hold, even if the objective function is strongly convex.

Theorem 4.6**.**

Consider the problem

[TABLE]

If the initial point is $({x}^{0},0,{w}^{0})$, or if ${w}^{k}=\rho$ for some $k$, then the ADMM sequence satisfies $({x}^{k},{y}^{k})\rightarrow(0,0)$ and ${w}^{k}\rightarrow-\infty$.

Even for linearly-constrained, convex, multiblock problems, this condition777For linear constraints $A_{1}x_{1}+\ldots+A_{n}x_{n}=b$, the equivalent statement is that $\operatorname{Im}(A_{n})\supseteq\bigcup_{i=1}^{n-1}\operatorname{Im}(A_{i})$. is close to indispensable. When all the other assumptions except 1.2 are satisfied, ADMM can still diverge if $\operatorname{Im}(Q)\not\supseteq\operatorname{Im}(A)$. In fact, [9, Thm 3.1] exhibits a simple 3-block convex problem with objective function $\phi\equiv 0$ on which ADMM diverges for any $\rho$. This condition is used explicitly [57, 33, 28] and implicitly [25] in other analyses of multiblock (nonconvex) ADMM.

4.2.3. Assumption 1.3

This assumption posits that the entire objective function $\phi$ is coercive on the feasible region, and imposes several conditions on the term $\psi(\mathcal{Z})$ for the final block $\mathcal{Z}$.

Let us first consider the conditions on $\psi$. The block $\mathcal{Z}$ is composed of three sub-blocks $Z_{0},Z_{1},Z_{2}$, and $\psi(\mathcal{Z})$ decomposes as $h(Z_{0})+g_{1}(Z_{S})+g_{2}(Z_{2})$, where $Z_{S}$ represents either $Z_{1}$ or $(Z_{0},Z_{1})$. There is a distinction between $Z_{0}$ and $\mathcal{Z}_{>}=(Z_{1},Z_{2})$: namely, $Z_{0}$ may be coupled with the other variables $\mathcal{X}$ in the nonlinear function $A$, whereas $\mathcal{Z}_{>}$ appears only in the linear function $Q(\mathcal{Z}_{>})$ which satisfies $\operatorname{Im}(Q)\supseteq\operatorname{Im}(A)$.

To understand the purpose of this assumption, consider the following ‘abstracted’ assumptions, which are implied by 1.3:

**M1: **

The objective is Lipschitz differentiable with respect to a ‘suitable’ subset of $\mathcal{Z}$.

**M2: **

ADMM yields sufficient decrease [2] when updating $\mathcal{Z}$. That is, for some ‘suitable’ subset ${\widetilde{\mathcal{Z}}}$ of $\mathcal{Z}$ and some $\epsilon>0$, we have $\mathcal{L}(\mathcal{X}^{+},\mathcal{Z},\mathcal{W})-\mathcal{L}(\mathcal{X}^{+},\mathcal{Z}^{+},\mathcal{W})\geq\epsilon\|{\widetilde{\mathcal{Z}}}-{\widetilde{\mathcal{Z}}}^{+}\|^{2}$.

A ‘suitable’ subset of $\mathcal{Z}$ is one whose associated images in the constraints satisfies 1.2. By design, our formulation $(P)$ uses the subset $\mathcal{Z}_{>}=(Z_{1},Z_{2})$ in this role. M1 follows from the fact that $g_{1},g_{2}$ are Lipschitz differentiable, and the other conditions in 1.3 are intended to ensure that M2 holds. For instance, the strong convexity assumption in 1.3(2) ensures that M2 holds with respect to $Z_{1}$ regardless of the properties of $Q_{1}$. The concept of sufficient decrease for descent methods is discussed in [2].

To connect this to the classical linearly-constrained problem, observe that an assumption corresponding to M1 is:

**AL: **

For the problem $(P1)$ (see page 1), $f_{n}(x_{n})$ is Lipschitz differentiable.

Thus, in this sense $\mathcal{Z}_{>}$ alone corresponds to the final block in the linearly-constrained case. In the multiaffine setting, we can add a sub-block $Z_{0}$ to the final block $\mathcal{Z}$, a nonsmooth term $h(Z_{0})$ to the objective function and a coupled constraint $A_{1}(\mathcal{X},Z_{0})$, but only to a limited extent: the interaction of the final block $\mathcal{Z}$ with these elements is limited to the variables $Z_{0}$.

As with 1.2, it would expand the scope of ADMM if AL, or the corresponding M1, could be relaxed. However, we find that for nonconvex problems, AL cannot readily be relaxed even in the linearly-constrained case, where AL is a standard assumption [57, 33, 28]. Furthermore, an example is given in [57, 36(a)] of a 2-block problem in which the function $f_{2}(x_{2})=\|x_{2}\|_{1}$ is nonsmooth, and it is shown that ADMM diverges for any $\rho$ when initialized at a given point. Thus, we suspect that AL/M1 is tight for general problems, though it may be possible to prove convergence for specific structured problems not satisfying M1.

M1 often has implications for modeling. When a constraint $C(\mathcal{X})=0$ fails to have the required structure, one can introduce a new slack variable $Z$, and replace that constraint by $C(\mathcal{X})-Z=0$, and add a term $g(Z)$ to the objective function to penalize $Z$. Because of M1, exact penalty functions such as $\lambda\|Z\|_{1}$ or the indicator function of $\{0\}$ fail to satisfy 1.3, so this reformulation is not exact. Based on the above discussion, this may be a limitation inherent to ADMM (as opposed to merely an artifact of existing proof techniques).

We turn now to M2. Note that AL corresponds only to M1, which is why 1.3 is more complicated than AL. There are two main sub-assumptions within 1.3 that ensure M2: that $g_{1}$ is strongly convex in $Z_{1}$, and the map $Q_{2}$ is injective. These assumptions are not tight888in the sense that this exact assumption is always necessary and cannot be replaced. since M2 may hold under alternative hypotheses. On the other hand, we are not aware of other assumptions that are as comparably simple and apply with the generality of 1.3; hence we have chosen to adopt the latter. For example, if we restrict the problem structure by assuming that the sub-block $Z_{0}$ is not present, then the condition that $g_{1}$ is strongly convex can be relaxed to the weaker condition that $\nabla^{2}g_{1}(Z_{1})+\rho Q_{1}^{T}Q_{1}\succeq mI$ for $m>0$. However, even in the absence of 1.3, one might show that specific problems, or classes of structured problems, satisfy the sufficient decrease property, using the general principles of ADMM outlined in Section 6.

Property M2 often arises implicitly when analyzing ADMM. In some cases, such as [25, 47], it follows either from strong convexity of the objective function, or because $A_{n}=I$ (and is thus injective). Proximal and majorized versions of ADMM are considered in [28, 33] and add quadratic terms which force M2 to be satisfied. The approach in [57], by contrast, takes a different approach and uses an abstract assumption which relates $\|A_{k}(x_{k}^{+}-x_{k})\|^{2}$ to $\|x_{k}^{+}-x_{k}\|^{2}$; in our experience, it is difficult to verify this abstract assumption in general, except when other properties such as strong convexity or injectivity of $A_{k}$ hold.

Finally, we remark on the coercivity of $\phi$ over the feasible region. It is common to assume coercivity (see, e.g. [33, 57]) to ensure that the sequence of iterates is bounded, which implies that limit points exist. In many applications, such as (DL) (Section 3.1), $\phi$ is independent of some of the variables. However, $\phi$ can still be coercive over the feasible region. For the variable-splitting formulation (DL3), this holds because of the constraints $X=X^{\prime}+X^{\prime\prime}$ and $Y=Y^{\prime}+Y^{\prime\prime}$. The objective function is coercive in $X^{\prime}$, $X^{\prime\prime}$, $Y^{\prime}$, and $Y^{\prime\prime}$, and therefore $X$ and $Y$ cannot diverge on the feasible region.

4.2.4. Assumption 2.1

The key element of this assumption is that $X_{0},\ldots,X_{n}$ may only be coupled by a Lipschitz differentiable function $F(X_{0},\ldots,X_{n})$, and the (possibly nonsmooth) terms $f_{0}(X_{0}),\ldots,f_{n}(X_{n})$ must be separable. This type of assumption is also used in previous works such as [12, 28, 57].

4.2.5. Assumption 2.2, 2.3

We have grouped 2.2, 2.3 together here because their motivation is the same. Our goal is to obtain conditions under which the convergence of the function differences $\mathcal{L}(\mathcal{X}_{<\ell}^{+},X_{\ell},\mathcal{X}_{>\ell},\mathcal{Z},\mathcal{W})-\mathcal{L}(\mathcal{X}_{<\ell}^{+},X_{\ell}^{+},\mathcal{X}_{>\ell},\mathcal{Z},\mathcal{W})$ implies that $\|X_{\ell}-X_{\ell}^{+}\|\rightarrow 0$ (and likewise for $Z_{0}$). This can be viewed as a much weaker analogue of the sufficient decrease property M2. In 2.2 and 2.3, we have presented several alternatives under which this holds. Under 2.2(1) and 2.3(1), the strengthened convexity condition (5.13), it is straightforward to show that

[TABLE]

(and likewise for $Z_{0}$), where $\Delta(t)$ is the [math]-forcing function arising from strengthened convexity. For 2.2(2) and 2.3(2), the inequality (4.4) holds with $\Delta(t)=at^{2}$, which is the sufficient decrease condition of [2]. Note that having $\Delta(t)\in O(t^{2})$ is important for proving convergence in the K-Ł setting, hence the additional hypotheses in Theorem 4.5.

As with 1.3, the assumptions in 2.2 and 2.3 are not tight, because (4.4) may occur under different conditions. We have chosen to use this particular set of assumptions because they are easily verifiable, and fairly general. The general results of Section 6 may be useful in analyzing ADMM for structured problems when the particular conditions of 2.2 are not satisfied.

4.2.6. Connection with proximal ADMM

When modeling, one may always ensure that 2.2(2a) is satisfied for $X_{\ell}$ by introducing a new variable $Z_{3}$ and a new constraint $X_{\ell}=Z_{3}$. This may appear to be a trivial reformulation of the problem, but it in fact promotes regularity of the ADMM subproblem in the same way as introducing a positive semidefinite proximal term.

Generalizing this trick, let $S$ be positive semidefinite, with square root $S^{1/2}$. Consider the constraint $\sqrt{\frac{2}{\rho}}S^{1/2}(X_{\ell}-Z_{3})=0$. The term of the augmented Lagrangian induced by this constraint is $\|X_{\ell}-Z_{3}\|_{S}^{2}$, where $\|\cdot\|_{S}$ is the seminorm $\|X\|_{S}^{2}=\langle X,SX\rangle$ induced by $S$. To see this, let $W_{0}$ be the Lagrange multiplier corresponding to this constraint.

Lemma 4.7**.**

If $W_{3}^{0}$ is initialized to [math], then for all $k\geq 1$, $Z_{3}^{k}=X_{\ell}^{k}$ and $W_{3}^{k}=0$. Consequently, the constraint $\sqrt{\frac{2}{\rho}}S^{1/2}(X_{\ell}-Z_{3})=0$ is equivalent to adding a proximal term $\|X_{\ell}-X_{\ell}^{k}\|^{2}_{S}$ to the minimization problem for $X_{\ell}$.

Note that proximal ADMM is often preferable to ADMM in practice [35, 19]. ADMM subproblems, which may have no closed-form solution because of the linear mapping in the quadratic penalty term, can often be transformed into a pure proximal mapping with a closed-form solution, by adding a suitable proximal term. Several applications of this approach are developed in [19]. Furthermore, for proximal ADMM, the conditions on $f_{i}$ in 2.2(2b) can be slightly weakened, by modifying Lemma 6.8 and Corollary 7.3 (see 6.10) to account for the proximal term as in [28].

5. Preliminaries

This section is a collection of definitions, terminology, and technical results which are not specific to ADMM. Proofs of the results in this section can be found in Appendix A, or in the provided references. The reader may wish to proceed directly to Section 6 and return here for details as needed.

5.1. General Subgradients and First-Order Conditions

In order to unify our treatment of first-order conditions, we use the notion of general subgradients, which generalize gradients and subgradients. When $f$ is smooth or convex, the set of general subgradients consists of the ordinary gradient or subgradients, respectively. Moreover, some useful functions that are neither smooth nor convex such as the indicator function of certain nonconvex sets possess general subgradients.

Definition 5.1**.**

Let $G$ be a closed and convex set. The tangent cone $T_{G}(x)$ of $G$ at the point $x\in G$ is the set of directions $T_{G}(x)=\operatorname{cl}(\{y-x:y\in G\})$. The normal cone $N_{G}(x)$ is the set $N_{G}(x)=\{v:\langle v,y-x\rangle\leq 0~{}\forall y\in G\}$.

Definition 5.2** ([45], 8.3).**

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\infty\}$ and $x\in\operatorname{dom}(f)$. A vector $v$ is a regular subgradient of $f$ at $x$, indicated by $v\in{\widehat{\partial}}f(x)$, if $f(y)\geq f(x)+\langle v,y-x\rangle+o(\|y-x\|)$ for all $y\in\mathbb{R}^{n}$. A vector $v$ is a general subgradient, indicated by $v\in\partial f(x)$, if there exist sequences $x_{n}\rightarrow x$ and $v_{n}\rightarrow v$ with $f(x_{n})\rightarrow f(x)$ and $v_{n}\in{\widehat{\partial}}f(x_{n})$. A vector $v$ is a horizon subgradient, indicated by $v\in\partial^{\infty}f(x)$, if there exist sequences $x_{n}\rightarrow x$, $\lambda_{n}\rightarrow 0$, and $v_{n}\in{\widehat{\partial}}f(x_{n})$ with $f(x_{n})\rightarrow f(x)$ and $\lambda_{n}v_{n}\rightarrow v$.

The properties of the general subgradient can be found in [45, §8].

Under the assumption that the objective function is proper and lower semicontinuous, the ADMM subproblems will satisfy a necessary first-order condition.

Lemma 5.3** ([45], 8.15).**

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\infty\}$ be proper and lower semicontinuous over a closed set $G\subseteq\mathbb{R}^{n}$. Let $x\in G$ be a point at which the following constraint qualification is fulfilled: the set $\partial^{\infty}f(x)$ of horizon subgradients contains no vector $v\neq 0$ such that $-v\in N_{G}(x)$. Then, for $x$ to be a local optimum of $f$ over $G$, it is necessary that $0\in\partial f(x)+N_{G}(x)$.

For our purposes, it suffices to note that when $G=\mathbb{R}^{n}$, the constraint qualification is trivially satisfied because $N_{G}(x)=\{0\}$. In the context of ADMM, this implies that the solution of each ADMM subproblem satisfies the first-order condition $0\in\partial\mathcal{L}$.

Problem $(P)$ has nonlinear constraints, and thus it is not guaranteed a priori that its minimizers satisfy first-order necessary conditions, unless a constraint qualification holds. However, Assumption 1 implies that the constant rank constraint qualification (CRCQ) [27, 42] is satisfied by $(P)$, and minimizers of $(P)$ will therefore satisfy first-order necessary conditions as long as the objective function is suitably regular. This follows immediately from 1.2 and the following lemma.

Lemma 5.4**.**

Let $C(x,z)=A(x)+Qz$, where $A(x)$ is smooth and $Q$ is a linear map with $\operatorname{Im}(Q)\supseteq\operatorname{Im}(A)$. Then for any points $x,z$, and any vector $w$, $(\nabla C(x,z))^{T}w=0$ if and only if $Q^{T}w=0$.

5.2. Multiaffine Maps

Every multiaffine map can be expressed as a sum of multilinear maps and a constant. This provides a useful concrete representation.

Lemma 5.5**.**

Let $\mathcal{M}(X_{1},\ldots,X_{n})$ be a multiaffine map. Then, $\mathcal{M}$ can be written in the form $\mathcal{M}(X_{1},\ldots,X_{n})=B+\sum_{j=1}^{m}\mathcal{M}_{j}(\mathcal{D}_{j})$ where $B$ is a constant, and each $\mathcal{M}_{j}(\mathcal{D}_{j})$ is a multilinear map of a subset $\mathcal{D}_{j}\subseteq(X_{1},\ldots,X_{n})$.

Let $\mathcal{M}(X_{1},\ldots,X_{n},Y)$ be multiaffine, with $Y$ a particular variable of interest, and $X=(X_{1},\ldots,X_{n})$ the other variables. By Lemma 5.5, grouping the multilinear terms $\mathcal{M}_{j}$ depending on whether $Y$ is one of the arguments of $\mathcal{M}_{j}$, we have

[TABLE]

where each $\mathcal{D}_{j}\subseteq(X_{1},\ldots,X_{n})$.

Definition 5.6**.**

Let $\mathcal{M}(X_{1},\ldots,X_{n},Y)$ have the structure (5.1). Let $\mathcal{F}_{Y}$ be the space of functions from $Y\rightarrow\operatorname{Im}(\mathcal{M})$. Let $\theta_{j}:\mathcal{D}_{j}\rightarrow\mathcal{F}_{Y}$ be the map999When $j>m_{1}$, $\theta_{j}(X)$ is a constant map of $Y$. given by $(\theta_{j}(X))(Y)=\mathcal{M}_{j}(\mathcal{D}_{j},Y)$. Here, we use the notation $\theta_{j}(X)$ for $\theta_{j}(\mathcal{D}_{j})$, with $\mathcal{D}_{j}$ taking the values in $X$. Finally, let $\mathcal{M}_{Y,X}=\sum_{j=1}^{m_{1}}\theta_{j}(X)$.

We call $\mathcal{M}_{Y,X}$ the $Y$-linear term of $\mathcal{M}$ (evaluated at $X$).

To motivate this definition, observe that when $X$ is fixed, the map $Y\mapsto\mathcal{M}(X,Y)$ is affine, with the linear component given by $\mathcal{M}_{Y,X}$ and the constant term given by $B_{X}=B+\sum_{j=m_{1}+1}^{m}\mathcal{M}_{j}(\mathcal{D}_{j})$. When analyzing the ADMM subproblem in $Y$, a multiaffine constraint $\mathcal{M}(X,Y)=0$ becomes the linear constraint $\mathcal{M}_{Y,X}(Y)=-B_{X}$.

The definition of multilinearity immediately shows the following.

Lemma 5.7**.**

$\theta_{j}$* is a multilinear map of $\mathcal{D}_{j}$. For every $X$, $\theta_{j}(X)$ is a linear map of $Y$, and thus $\mathcal{M}_{Y,X}$ is a linear map of $Y$.*

Example**.**

Consider $\mathcal{M}(X_{1},X_{2},X_{3},X_{4})=X_{1}X_{2}X_{3}+X_{2}X_{3}X_{4}+X_{2}+B=0$ for square matrices $X_{1},X_{2},X_{3},X_{4}$. Taking $Y=X_{3}$ as the variable of focus, and $X=(X_{1},X_{2},X_{4})$, we have $(\theta_{1}(X))(Y)=X_{1}X_{2}Y$, $(\theta_{2}(X))(Y)=X_{2}YX_{4}$, $(\theta_{3}(X))(Y)=X_{2}$, $(\theta_{4}(X))(Y)=B$, and thus $\mathcal{M}_{Y,X}$ is the linear map $Y\mapsto X_{1}X_{2}Y+X_{2}YX_{4}$.

Our general results in Section 6 require smooth constraints, which holds for multiaffine maps.

Lemma 5.8**.**

Multiaffine maps are smooth, and in particular, biaffine maps are Lipschitz differentiable.

5.3. Smoothness, Convexity, and Coercivity

Definition 5.9**.**

A function $g$ is $M$-Lipschitz differentiable if $g$ is differentiable and its gradient is Lipschitz continuous with modulus $M$, i.e. $\|\nabla g(x)-\nabla g(y)\|\leq M\|x-y\|$ for all $x,y$.

A function $g$ is $(m,M)$-strongly convex if $g$ is convex, $M$-Lipschitz differentiable, and satisfies $g(y)\geq g(x)+\langle\nabla g(x),y-x\rangle+\frac{m}{2}\|y-x\|^{2}$ for all $x$ and $y$. The condition number of $g$ is $\kappa:=\frac{M}{m}$.

Lemma 5.10**.**

If $g$ is $M$-Lipschitz differentiable, then $|g(y)-g(x)-\langle\nabla g(x),y-x\rangle|\leq\frac{M}{2}\|y-x\|^{2}$.

Lemma 5.11**.**

If $g(\cdot,\cdot)$ is $M$-Lipschitz differentiable, then for any fixed $y$, the function $h_{y}(\cdot)=g(\cdot,y)$ is $M$-Lipschitz differentiable. If $g(\cdot,\cdot)$ is $(m,M)$-strongly convex, then $h_{y}(\cdot)$ is $(m,M)$-strongly convex.

Definition 5.12**.**

A function $\phi$ is said to be coercive on the set $\Omega$ if for every sequence $\{x_{k}\}_{k=1}^{\infty}\subseteq\Omega$ with $\|x_{k}\|\rightarrow\infty$, then $\phi(x_{k})\rightarrow\infty$.

Definition 5.13**.**

A function $\Delta:\mathbb{R}\rightarrow\mathbb{R}$ is 0-forcing if $\Delta\geq 0$, and any sequence $\{t_{k}\}$ has $\Delta(t_{k})\rightarrow 0$ only if $t_{k}\rightarrow 0$. A function $f$ is said to satisfy a strengthened convexity condition if there exists a [math]-forcing function $\Delta$ such that for any $x,y$, and any $v\in\partial f(x)$, $f$ satisfies

[TABLE]

Remark 5.14**.**

The strengthened convexity condition is stronger than convexity, but weaker than strong convexity. An example is given by higher-order polynomials of degree $d$ composed with the Euclidean norm, which are not strongly convex for $d>2$. An example is the function $f(x)=\|x\|_{2}^{3}$, which is not strongly convex but does satisfy the strengthened convexity condition. This function appears in the context of the cubic-regularized Newton method [59]. We note that ADMM can be applied to solve the nonconvex cubic-regularized Newton subproblem

[TABLE]

by performing the splitting $\min\limits_{x,y}q(x)+h(y)$ for $q(x)=\frac{1}{2}x^{T}Ax+b^{T}x$ and $h(y)=\frac{\mu}{3}\|x\|_{2}^{3}$.

Since we will subsequently show (Theorem 4.1) that the sequence of ADMM iterates is bounded, the strengthened convexity condition can be relaxed. It would be sufficient to assume that for every compact set $G$, a 0-forcing function $\Delta_{G}$ exists so that (5.2) holds with $\Delta_{G}$ whenever $x,y\in G$.

5.4. Distances and Translations

Definition 5.15**.**

For a symmetric matrix $S$, let $\lambda_{min}(S)$ be the minimum eigenvalue of $S$, and let $\lambda_{++}(S)$ be the minimum positive eigenvalue of $S$.

Lemma 5.16**.**

Let $R$ be a matrix and $y\in\operatorname{Im}(R)$. Then $\|y\|^{2}\leq\lambda_{++}^{-1}(R^{T}R)\|R^{T}y\|^{2}$.

Lemma 5.17**.**

Let $A$ be a matrix, and $b,c\in\operatorname{Im}(A)$. There exists a constant $\alpha_{A}$ with $\operatorname{dist}(\{x:Ax=b\},\{x:Ax=c\})\leq\alpha_{A}\|b-c\|$. Furthermore, we may take $\alpha_{A}\leq\sqrt{\lambda_{++}^{-1}(AA^{T})}$.

Lemma 5.18**.**

Let $g$ be a $(m,M)$-strongly convex function with condition number $\kappa=\frac{M}{m}$, let $\mathcal{C}$ be a closed and convex set with $\mathcal{C}_{1}=a+\mathcal{C}$ and $\mathcal{C}_{2}=b+\mathcal{C}$ being two translations of $\mathcal{C}$, let $\delta=\|b-a\|$, and let $x^{\ast}=\operatorname{argmin}\{g(x):x\in\mathcal{C}_{1}\}$ and $y^{\ast}=\operatorname{argmin}\{g(y):y\in\mathcal{C}_{2}\}$. Then, $\|x^{\ast}-y^{\ast}\|\leq(1+2\kappa)\delta$.

Lemma 5.19**.**

Let $h$ be a $(m,M)$-strongly convex function, $A$ a linear map of $x$, and $\mathcal{C}$ a closed and convex set. Let $b_{1},b_{2}\in\operatorname{Im}(A)$, and consider the sets $\mathcal{U}_{1}=\{x:Ax+b_{1}\in\mathcal{C}\}$ and $\mathcal{U}_{2}=\{x:Ax+b_{2}\in\mathcal{C}\}$, which we assume to be nonempty. Let $x^{\ast}=\operatorname{argmin}\{h(x):x\in\mathcal{U}_{1}\}$ and $y^{\ast}=\operatorname{argmin}\{h(y):y\in\mathcal{U}_{2}\}$. Then, there exists a constant $\gamma$, depending on $\kappa$ and $A$ but independent of $\mathcal{C}$, such that $\|x^{\ast}-y^{\ast}\|\leq\gamma\|b_{2}-b_{1}\|$.

5.5. K-Ł Functions

Definition 5.20**.**

Let $f$ be proper and lower semicontinuous. The domain $\operatorname{dom}(\partial f)$ of the general subgradient mapping is the set $\{x:\partial f(x)\neq\emptyset\}$.

Definition 5.21** ([2], 2.4).**

A function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}\cup\{\infty\}$ is said to have the Kurdyka-Łojasiewicz (K-Ł) property at $x\in\operatorname{dom}(\partial f)$ if there exist $\eta\in(0,\infty]$, a neighborhood $U$ of $x$, and a continuous concave function $\varphi:[0,\eta)\rightarrow\mathbb{R}$ such that:

1. (1)

$\varphi(0)=0$** 2. (2)

$\varphi$* is smooth on $(0,\eta)$* 3. (3)

For all $s\in(0,\eta)$, $\varphi^{\prime}(s)>0$ 4. (4)

For all $y\in U\cap\{w:f(x)<f(w)<f(x)+\eta\}$, the Kurdyka-Łojasiewicz inequality holds:

[TABLE]

A proper, lower semicontinuous function $f$ that satisfies the K-Ł property at every point of $\operatorname{dom}(\partial f)$ is called a K-Ł function.

A large class of K-L functions is provided by the semialgebraic functions, which include many functions of importance in optimization.

Definition 5.22** ([2], 2.1).**

A subset $S$ of $\mathbb{R}^{n}$ is (real) semialgebraic if there exists a finite number of real polynomial functions $P_{ij},Q_{ij}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ such that

[TABLE]

A function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ is semialgebraic if its graph $\{(x,y)\in\mathbb{R}^{n+m}:f(x)=y\}$ is a real semialgebraic subset of $\mathbb{R}^{n+m}$.

The set of semialgebraic functions is closed under taking finite sums and products, scalar products, and composition. The indicator function of a semialgebraic set is a semialgebraic function, as is the generalized inverse of a semialgebraic function. More examples can be found in [1].

The key property of K-Ł functions is that if a sequence $\{x^{k}\}_{k=0}^{\infty}$ is a ‘descent sequence’ with respect to a K-Ł function, then limit points of $\{x^{k}\}$ are necessarily unique. This is formalized by the following;

Theorem 5.23** ([2], 2.9).**

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be a proper and lower semicontinuous function. Consider a sequence $\{x^{k}\}_{k=0}^{\infty}$ satisfying the properties:

**H1: **

There exists $a>0$ such that for each $k$, $f(x^{k+1})-f(x^{k})\leq-a\|x^{k+1}-x^{k}\|^{2}$.

**H2: **

There exists $b>0$ such that for each $k$, there exists $w^{k+1}\in\partial f(x^{k+1})$ with $\|w^{k+1}\|\leq b\|x^{k+1}-x^{k}\|$.

If $f$ is a K-Ł function, and $x^{\ast}$ is a limit point of $\{x^{k}\}$ with $f(x^{k})\rightarrow f(x^{\ast})$, then $x^{k}\rightarrow x^{\ast}$.

6. General Properties of ADMM

In this section, we will derive results that are inherent properties of ADMM, and require minimal conditions on the structure of the problem. We first work in the most general setting where $C$ in the constraint $C(U_{0},\ldots,U_{n})=0$ may be any smooth function, the objective function $f(U_{0},\ldots,U_{n})$ is proper and lower semicontinuous, and the variables $\{U_{0},\ldots,U_{n}\}$ may be coupled. We then specialize to the case where the constraint $C(U_{0},\ldots,U_{n})$ is multiaffine, which allows us to quantify the changes in the augmented Lagrangian using the subgradients of $f$. Finally, we specialize to the case where the objective function splits into $F(U_{0},\ldots,U_{n})+\sum_{i=0}^{n}g_{i}(U_{i})$ for a smooth coupling function $F$, which allows finer quantification using the subgradients of the augmented Lagrangian.

The results given in this section hold under very weak conditions; hence, these results may be of independent interest, as tools for analyzing ADMM in other settings.

6.1. General Objective and Constraints

In this section, we consider

[TABLE]

The augmented Lagrangian is given by

[TABLE]

and ADMM performs the updates as in Algorithm 1. We assume only the following.

Assumption 3**.**

The following hold.

A 3.1**.**

For sufficiently large $\rho$, every ADMM subproblem attains its optimal value.

A 3.2**.**

$C(U_{0},\ldots,U_{n})$* is smooth.*

A 3.3**.**

$f(U_{0},\ldots,U_{n})$* is proper and lower semicontinuous.*

This assumption ensures that the $\operatorname{argmin}$ in Algorithm 1 is well-defined, and that the first-order condition in Lemma 5.3 holds at the optimal point. The results in this section are extensions of similar results for ADMM in the classical setting (linear constraints, separable objective function), so it is interesting that the ADMM algorithm retains many of the same properties under the generality of Assumption 3. We defer the proofs to the appendix.

Lemma 6.1**.**

Let $\mathcal{U}=(U_{0},\ldots,U_{n})$ denote the set of all variables. The ADMM update of the dual variable $W$ increases the augmented Lagrangian such that $\mathcal{L}(\mathcal{U}^{+},W^{+})-\mathcal{L}(\mathcal{U}^{+},W)=\rho\|C(\mathcal{U}^{+})\|^{2}=\frac{1}{\rho}\|W-W^{+}\|^{2}$. If $\|W-W^{+}\|\rightarrow 0$, then $\nabla_{W}\mathcal{L}({\mathcal{U}}^{k},{W}^{k})\rightarrow 0$