Convergence and Applications of ADMM on the Multi-convex Problems

TL;DR

This paper introduces a generic ADMM framework for multi-convex problems, providing convergence guarantees and demonstrating effectiveness through real-world experiments.

Contribution

It proposes a new ADMM framework with convergence guarantees for multi-convex problems, extending its applicability and providing theoretical and empirical validation.

Findings

Proven convergence to a Nash point with sublinear rate

Effective on ten real-world datasets

Demonstrates scalability and robustness

Abstract

In recent years, although the Alternating Direction Method of Multipliers (ADMM) has been empirically applied widely to many multi-convex applications, delivering an impressive performance in areas such as nonnegative matrix factorization and sparse dictionary learning, there remains a dearth of generic work on proposed ADMM with a convergence guarantee under mild conditions. In this paper, we propose a generic ADMM framework with multiple coupled variables in both objective and constraints. Convergence to a Nash point is proven with a sublinear convergence rate $o(1/k)$. Two important applications are discussed as special cases under our proposed ADMM framework. Extensive experiments on ten real-world datasets demonstrate the proposed framework's effectiveness, scalability, and convergence properties. We have released our code at \url{https://github.com/xianggebenben/miADMM}.

Tables

Table 1. Table 1: Performance in Experiment I.

Mean
Method	MSE	MSLE	MAE	EV	R2
JFS	114.1052	0.4531	8.4349	0.2948	0.2948
CMTL	114.9892	0.4647	8.4756	0.2876	0.2875
mtLasso	115.3143	0.4625	8.4725	0.2873	0.2873
cASO	137.8336	0.5204	9.3450	0.1606	0.1605
BCD	149.2313	0.5577	9.8057	0.1299	0.0777
ADMM(${\lambda }_1\left(1\right)$)	113.6975	0.4423	8.4024	0.2950	0.2960
ADMM(${\lambda }_1\left(2\right)$)	113.2400	0.4428	8.3943	0.3002	0.3002
Standard Deviation
Method	MSE	MSLE	MAE	EV	R2
JFS	2.02	0.02	0.06	0.02	0.02
CMTL	1.85	0.02	0.05	0.01	0.01
mtLasso	1.77	0.02	0.05	0.01	0.01
cASO	7.26	0.01	0.06	0.01	0.01
BCD	1.41	0.01	0.06	0.15	0.01
ADMM(${\lambda }_1\left(1\right)$)	0.83	0.005	0.03	0.01	0.01
ADMM(${\lambda }_1\left(2\right)$)	0.95	0.01	0.04	0.02	0.02

Table 2. Table 2: Event forecasting performance in AUC in each of the 9 datasets.

	BR	CL	CO	EC	EL	MX	PY	UY	VE
LogReg	0.686	0.677	0.644	0.599	0.618	0.661	0.616	0.628	0.667
LASSO	0.685	0.677	0.648	0.603	0.636	0.665	0.615	0.666	0.669
MTL	0.722	0.669	0.810	0.617	0.772	0.795	0.600	0.811	0.771
MREF	0.714	0.563	0.515	0.784	0.612	0.693	0.658	0.681	0.588
DHML	0.845	0.683	0.846	0.839	0.780	0.793	0.737	0.835	0.835
BCD	0.847	0.668	0.850	0.830	0.773	0.800	0.736	0.835	0.856
ADMM $\left({\lambda }_2\left(1\right)\right)$	0.864	0.699	0.870	0.848	0.794	0.820	0.746	0.850	0.867
ADMM $\left({\lambda }_2\left(2\right)\right)$	0.867	0.701	0.872	0.851	0.798	0.823	0.747	0.847	0.865

Table 3. Table 3: Comparison of running time (in seconds) on 9 datasets in Experiment II.

	LogReg	LASSO	MTL	MREF	DHML	ADMM
BR	30193	1535	233	25889	332	14
CL	2981	242	35	6521	852	11
CO	8060	780	108	14714	87	31
EC	312	295	17	4332	46	25
EL	551	261	17	4669	33	3
MX	17712	2043	853	31349	175	29
PY	7297	527	40	9495	242	5
UY	748	336	20	5305	82	3
VE	5563	1008	49	5769	179	28

Equations

$${\min }_{x,z}f(x)+g(z)s.t.Ax+Bz=c.$$

$$\Vert \nabla G_1(x^{{}^{\prime }})-\nabla G_1(x^{{}^{\prime \prime }})\Vert \le D\Vert x^{{}^{\prime }}-x^{{}^{\prime \prime }}\Vert .$$

$$G_4(x^{{}^{\prime \prime }})\ge G_4(x^{{}^{\prime }})+(v^{{}^{\prime }}{)}^T(x^{{}^{\prime \prime }}-x^{{}^{\prime }})+(H/2)\Vert x^{{}^{\prime \prime }}-x^{{}^{\prime }}{\Vert }_2^2.$$

$$G_5(a_1^{\ast },\cdots ,a_{i-1}^{\ast },a_i^{\ast },a_{i+1}^{\ast },\cdots ,a_m^{\ast })\le G_5(a_1^{\ast },\cdots ,a_{i-1}^{\ast },a_i,a_{i+1}^{\ast },\cdots ,a_m^{\ast }),$$

$$\forall (a_1^{\ast },\cdots ,a_{i-1}^{\ast },a_i,a_{i+1}^{\ast },\cdots ,a_m^{\ast })\in dom(G_5),(i=1,\cdots ,m).$$

$${\min }_{x_1,\cdots ,x_n,z}F(x_1,\cdots ,x_n,z)=f(x_1,\cdots ,x_n)+h(z)s.t.{\sum }_{i=1}^n{A}_i{x}_i-z=0.$$

$$L_{\rho }(x_1,\cdots ,x_n,z,y)$$

$$x_i^{k+1}\leftarrow \arg {\min }_{x_i}f(\cdots ,x_{i-1}^{k+1},x_i,x_{i+1}^k,\cdots )+(y^k{)}^T{A}_i{x}_i$$

$$+(\rho /2)\Vert {\sum }_{j=1}^{i-1}{A}_j{x}_j^{k+1}+A_i{x}_i+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k{\Vert }_2^2.$$

$$z^{k+1}\leftarrow \arg {\min }_z{L}_{\rho }(\cdots ,x_n^{k+1},z,y^k)$$

$$=\arg {\min }_{z}h(z)-(y^k{)}^{T}z+(\rho /2)\Vert {\sum }_{i=1}^n{A}_i{x}_i^{k+1}-z{\Vert }_2^2.$$

$$L_{\rho }(x_1^k,\cdots ,x_n^k,z^k,y^k)-L_{\rho }(x_1^{k+1},\cdots ,x_n^{k+1},z^{k+1},y^{k+1})$$

$$\ge C_2(\Vert z^{k+1}-z^k{\Vert }_2^2+{\sum }_{i=1}^n\Vert A_i(x_i^{k+1}-x_i^k){\Vert }_2^2).$$

$$\sum A_i{x}_i^{\ast }-z^{\ast }=0.\text{(feasibility)}$$

$$F(x_1^{\ast },\cdots ,x_n^{\ast },z^{\ast })\le F(x_1^{\ast },\cdots ,x_{i-1}^{\ast },x_i,x_{i+1}^{\ast },\cdots ,x_n^{\ast },z^{\ast }),$$

$$\forall (x_1^{\ast },\cdots ,x_{i-1}^{\ast },x_i,x_{i+1}^{\ast },\cdots ,x_n^{\ast },z^{\ast })\in dom(F),(i=1,\cdots ,n).$$

$$F(x_1^{\ast },\cdots ,x_n^{\ast },z^{\ast })\le F(x_1^{\ast },\cdots ,x_n^{\ast },z)\forall (x_1^{\ast },\cdots ,x_n^{\ast },z)\in dom(F)\text{(Nash point)}.$$

$${\min }_{w_1,\cdots ,w_n}{\sum }_{i=1}^n(Los{s}_i(w_i)+{\Omega }_i(w_i))$$

$$s.t.w_{i,j}{w}_{i+1,j}\ge 0(i=1,2,\cdots ,n-1,j=1,2,\cdots ,m).$$

$${\min }_{w_1,\cdots ,w_n,z}{\sum }_{i=1}^n(Los{s}_i(w_i)+{\Omega }_i(z_i))+{\lambda }_1{\sum }_{i=1}^{n-1}{\sum }_{j=1}^m{c}_1(w_{i,j}{w}_{i+1,j})$$

$$s.t.z_i=w_i(i=1,2,\cdots ,n).$$

$${\min }_{{\beta }_1,\cdots ,{\beta }_n}Loss({\beta }_1,\cdots ,{\beta }_n)+{\sum }_{i=1}^n{\omega }_i({\beta }_i)$$

$$s.t.\exists ({\beta }_i,{\beta }_j)\in E_s,\exists ({\beta }_p,{\beta }_q)\in E_d(1\le i,j,p,q\le n).$$

$${\min }_{{\beta }_1,\cdots ,{\beta }_n,z}Loss({\beta }_1,\cdots ,{\beta }_n)+{\sum }_{i=1}^n{\omega }_i(z_i)+{\lambda }_2({\sum }_{({\beta }_i,{\beta }_j)\in E_s}{c}_2({\beta }_i,{\beta }_j)$$

$$+{\sum }_{({\beta }_p,{\beta }_q)\in E_d}{c}_3({\beta }_p,{\beta }_q))s.t.z_i={\beta }_i(i=1,2,\cdots ,n)$$

$$h(z_1)\le h(z_2)+\nabla h(z_2{)}^T(z_1-z_2)+(H/2)\Vert z_1-z_2{\Vert }^2,-h(z_1)\le -h(z_2)-\nabla h(z_2{)}^T(z_1-z_2)+(H/2)\Vert z_1-z_2{\Vert }^2.$$

$$\nabla h(z^k)-y^{k-1}-\rho ({\sum }_{i=1}^n{A}_i{x}_i^k-z^k)=0.$$

$$L_{\rho }(\cdots ,x_{i-1}^{k+1},x_i^k,\cdots )-L_{\rho }(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )\ge (\rho /2)\Vert A_i{x}_i^k-A_i{x}_i^{k+1}{\Vert }_2^2.$$

$$L_{\rho }(\cdots ,x_{i-1}^{k+1},x_i^k,\cdots )-L_{\rho }(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )$$

$$=f(\cdots ,x_{i-1}^{k+1},x_i^k,\cdots )-f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )$$

$$+(y^k{)}^T(A_i{x}_i^k-A_i{x}_i^{k+1})+(\rho /2)\Vert {\sum }_{j=1}^{i-1}{A}_j{x}_j^{k+1}+{\sum }_{j=i}^n{A}_j{x}_j^k-z^k{\Vert }_2^2-(\rho /2)\Vert {\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k{\Vert }_2^2$$

$$=f(\cdots ,x_{i-1}^{k+1},x_i^k,\cdots )-f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )$$

$$+(y^k{)}^T(A_i{x}_i^k-A_i{x}_i^{k+1})+(\rho /2)\Vert A_i{x}_i^k-A_i{x}_i^{k+1}{\Vert }_2^2+\rho ({\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k{)}^T(A_i{x}_i^k-A_i{x}_i^{k+1})$$

$$=f(\cdots ,x_{i-1}^{k+1},x_i^k,\cdots )-f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )$$

$$+(A_i^T{y}^k+\rho A_i^T({\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k){)}^T(x_i^k-x_i^{k+1})+(\rho /2)\Vert A_i{x}_i^k-A_i{x}_i^{k+1}{\Vert }_2^2.$$

$$0\in {\partial }_{x_i}{L}_{\rho }(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )$$

$$={\partial }_{x_i}f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )+A_i^T{y}^k+\rho A_i^T({\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k)$$

$$-A_i^T{y}^k-\rho A_i^T({\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k)\in {\partial }_{x_i}f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots ).$$

$$f(\cdots ,x_{i-1}^{k+1},x_i^k,\cdots )$$

$$\ge f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )+(-A_i^T{y}^k-\rho A_i^T({\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k){)}^T(x_i^{k+1}-x_i^k)$$

$$=f(\cdots ,x_i^{k+1},x_{i+1}^k,\cdots )+(A_i^T{y}^k+\rho A_i^T({\sum }_{j=1}^i{A}_j{x}_j^{k+1}+{\sum }_{j=i+1}^n{A}_j{x}_j^k-z^k){)}^T(x_i^k-x_i^{k+1}).$$

Full text

1 Introduction

Due to the advantages and popularity of non-differentiable regularized and distributive computing for complex optimization problems, the Alternating Direction Method of Multipliers (ADMM) has received a great deal of attention in recent years [6]. The standard ADMM was originally proposed to solve the following separable convex optimization problem:

[TABLE]

where $f(x)$ and $g(z)$ are closed convex functions, $A$ and $B$ are matrices and $c$ is a vector. There are extensive reports in the literature exploring the theoretical properties for convex optimization problems related to ADMM and its variants, including multi-block ADMM [12], Bregman ADMM [30], fast ADMM [14, 18], and stochastic ADMM [24]. ADMM has now been extended to cover a wide range of nonconvex problems, and it has achieved outstanding performance in many practical applications [40].

Unlike convex problems, the convergence theory on the nonconvex ADMM is much more difficult, and considerable progress has been made on this problem, please refer to Section 2 for a detailed summary. Recently, however, there has been an increasing number of real-world applications where the objective functions are multi-convex (i.e. nonconvex for all the variables but convex for each when all the others are fixed). For example, nonnegative matrix factorization, which aims to decompose a matrix into a product of two matrices, has been applied widely in computer vision, machine learning, and various other fields [19]; A bilinear matrix inequality problem has been designed for the analysis of linear and nonlinear uncertain systems [16].

All of the above applications can be considered as special cases of multi-convex optimization problems. However, such problems have not yet been rigorously and systematically investigated by ADMM. Moreover, the convergence properties of the ADMM required to solve such problems remain unknown. In this work, we propose mild conditions to ensure the convergence of ADMM to a Nash point on the multi-convex problems with a sublinear convergence rate $o(1/k)$. We also discuss how our ADMM is applied to two important applications. Extensive experiments show the effectiveness of our proposed ADMM. Our contributions in this paper include:

• •

We propose an ADMM framework to solve the multi-convex problem, and we investigate the convergence properties of the proposed ADMM. Specifically, we prove that the objective value and the residual are convergent. Moreover, any limit point generated by the proposed ADMM is a Nash point of the original problem. The convergence rate of the proposed ADMM is $o(1/k)$.

• •

We demonstrate two important and promising applications that are special cases of our proposed ADMM framework and benefit from its theoretical properties. Specifically, we show how these applications can be transformed equivalently to fit into the proposed ADMM framework.

• •

We conduct extensive experiments to validate our proposed ADMM. Experiments on ten real-world datasets demonstrate its effectiveness, scalability, and convergence properties.

The rest of this paper is summarized as follows: Section 2 summarizes previous work related to this paper. Section 3 introduces the ADMM algorithm and its convergence properties. In Section 4, the proposed ADMM algorithm is applied to several important applications. Extensive experiments are described in Section 5. The paper concludes with a summary of the work in Section 6.

2 Related Work

Multi-convex optimization problems: Some works studied multi-convex problems. The earliest work required that the objective function was differentiable continuous and strictly convex [38]. Various conditions on separability and regularity on the objective functions have been discussed in [28, 29]. In the most recent work, Xu and Yin presented three types of multi-convex algorithms and analyzed convergence based on either Lipschitz differentiability or strong convexity assumption [39]. For a comprehensive survey, please refer to [26].

Convergence analysis of ADMM: Existing literature on the convergence analysis of ADMM can be categorized into two classes: the convex ADMM and the nonconvex ADMM. The convex ADMM is investigated relatively well compared with the nonconvex ADMM. Existing works either study suitable stepsizes of the convex ADMM or extend ADMM to the stochastic version. For example, Bai et al. proposed a generalized symmetric ADMM to solve the multi-block separable objective by updating the Lagrange multiplier twice with suitable stepsizes [3]; Gu et al. extended contractive Peaceman-Rachford splitting method to ADMM with larger stepsizes [15]; Ouyang et al. proposed a stochastic ADMM with a convergence rate $O(\frac{1}{\sqrt{t}})$. Despite the outstanding performance of the nonconvex ADMM, its convergence theory is not well established due to the complexity of both coupled objectives and various (inequality and equality) constraints. Most existing works discussed the convergence of the nonconvex ADMM on separable objectives: they provided convergence guarantee to the stationary solutions with different assumptions [5, 9, 10, 20]. Some works explored more difficult cases where the objectives are coupled: for example, Wang et al. presented mild convergence conditions of the nonconvex ADMM where the objective can be nonsmooth [37]; Gao et al. explored the convergence condition of ADMM on multi-affine constraints [13]; Wang et al. gave the convergence proofs of ADMM in the nonconvex deep learning problems [31, 33, 34]; while experiments by Wang and Zhao showed that the ADMM was not necessarily convergent in the nonlinear-constrained problems [35].

3 ADMM on the Multi-convex Problems

In this section, we present an ADMM framework to solve Problem 1.

3.1 Preliminaries

First, the definition of Lipschitz differentiability is shown as follows [8]:

Definition 1 (Lipschitz Differentiability)

Any arbitrary differentiable function $G_{1}:\mathbb{R}^{m}\rightarrow\mathbb{R}$ is Lipschitz differentiable if for any $x^{{}^{\prime}},x^{{}^{\prime\prime}}\in\mathbb{R}^{m}$,

[TABLE]

where $D\geq 0$ is constant and $\nabla G_{1}(x)$ denotes the gradient of $G_{1}(x)$.

The following defines strong convexity, which is indispensable for the proof of convergence to a Nash point.

Definition 2 (Strong Convexity)

A convex function $G_{4}(x)$ is strongly convex if there exists $H>0$ such that for $\forall x^{{}^{\prime}},x^{{}^{\prime\prime}}\in dom(G_{4})$, the following holds

[TABLE]

where $\forall v^{{}^{\prime}}\in\partial G_{4}(x^{{}^{\prime}})$ is a subdifferential of $G_{4}$ at $x^{{}^{\prime}}$.

Finally, the Nash point is defined as follows [39]:

Definition 3 (Nash Point)

Given $G_{5}(a_{1},a_{2},\cdots,a_{m})$, a Nash point $(a^{*}_{1},a^{*}_{2},\cdots,a^{*}_{m})$ satisfies the following property:

[TABLE]

Naturally, when we optimize one variable while fixing others, the Nash point ensures the optimality of this variable [39]. Without loss of generality, we assume that Problem 1 has at least a Nash point, and in the next section, we will prove that any limit point generated by ADMM converges to a Nash point.

3.2 The ADMM algorithm

The following problem is our focus in this paper:

Problem 1

[TABLE]

where $x_{i}\in\mathbb{R}^{p_{i}}(i=1,\cdots,n)$, $z\in\mathbb{R}^{q}$, $f(x_{1},\cdots,x_{n}):\mathbb{R}^{p}\rightarrow\mathbb{R}\cup\{\infty\}(p=\sum\nolimits_{i=1}^{n}p_{i})$ are proper, continuous, multi-convex and possibly nonsmooth functions, $h(z)$ is a proper, differentiable and convex function. $A_{i}\in\mathbb{R}^{q\times p_{i}}(i=1,\cdots,n)$ are matrices. Obviously, the domain of $F$ is $dom(F)=\{(x_{1},\cdots,x_{n},z)|\ \sum\nolimits_{i=1}^{n}A_{i}x_{i}-z=0\}$. Without the loss of generality, the objective of Problem 1 is assumed to be bounded from below.

To ensure the convergence of the proposed ADMM, some mild assumptions are imposed, which are shown as follows:

Assumption 1** (Lipschitz Differentiability)**

$h(z)$* is Lipschitz differentiable with constant $H\geq 0$.*

Most loss functions such as the cross-entropy loss and the square loss are Lipschitz differentiable [34]. In order to propose the ADMM algorithm, the augmented Lagrangian function can be formulated mathematically as follows:

[TABLE]

where $y$ is a dual variable and $\rho>0$ is a penalty parameter. The proposed ADMM aims to optimize the following $n+1$ subproblems alternately.

[TABLE]

Algorithm 1 is presented for Problem 1. Concretely, Lines 3-5 and 6 update $x^{k+1}_{i}(i=1,\cdots,n)$ and $z^{k+1}$, respectively. Line 7 updates the primal residual $r^{k+1}$, which is defined in accordance with the standard ADMM [6]: it measures how the linear constraint $\sum\nolimits_{i=1}^{n}A_{i}x_{i}-z=0$ is violated. Line 8 updates the dual variable $y^{k+1}$, which follows the routine of the standard ADMM. Line 10 uses the norm of the primal residual $r$ as a condition to terminate the algorithm, where $\delta>0$ is a threshold. Each subproblem is convex and implicitly assumed to be solvable.

3.3 Convergence Analysis

This section focuses on the convergence of the proposed ADMM algorithm. Specifically, the first lemma states that the augmented Lagrangian $L_{\rho}$ keeps decreasing, which is stated as follows.

Lemma 1 (Objective Descent)

If $\rho>2H$ so that $C_{1}=\rho/2-H/2-H^{2}/\rho>0$, then there exists $C_{2}=\min(\rho/2,C_{1})$ such that

[TABLE]

Lemma 1 holds under Assumption 1, and its proof can be found in Section 0.B in the supplementary materials111The supplementary materials are available at https://github.com/xianggebenben/miADMM/blob/main/multi_convex_ADMM-13-18.pdf. The next lemma states that the augmented Lagrangian is bounded from below, as shown below:

Lemma 2 (Objective Bound)

If $\rho>2H$, then $L_{\rho}(x_{1}^{k},\cdots,x^{k}_{n},z^{k},y^{k})$ is lower bounded.

The proof of Lemma 2 can be found in Section 0.B in the supplementary materials 1. Now we can prove that the proposed ADMM converges globally in the following theorem.

Theorem 3.1 (Residual and Objective Convergence)

*If $\rho>2H$, then for the bounded sequence $(x_{1}^{k},\cdots,x_{n}^{k},z^{k},y^{k})$, then it has the following properties:

a). Residual convergence. This means that as $k\rightarrow\infty$, $r^{k}\rightarrow 0$, where $r^{k}$ is defined in Algorithm 1.

b). Objective convergence. This means that as $k\rightarrow\infty$, $F(x^{k}_{1},\cdots,x^{k}_{n},z^{k})$ converges.*

Theorem 3.1 guarantees the convergence of the proposed ADMM, whose proof is in Section 0.C in the supplementary materials 1. However, $x^{k}_{i}(i=1,\cdots,n)$ and $z^{k}$ are not necessarily shown to be convergent. The next theorem states that any limit point is a feasible Nash Point of Problem 1.

Theorem 3.2 (Convergence to a Nash Point)

Let $\rho>2H$, if either of two assumptions hold: (a). $A_{i}(i=1,\cdots,n)$ have full rank. (b). $F$ is strongly convex with regard to $x_{i}$. Then for bounded variables $(x^{k}_{1},\cdots,x^{k}_{n},z^{k})$, it has at least a limit point $(x^{*}_{1},\cdots,x^{*}_{n},z^{*})$, and any limit point $(x^{*}_{1},\cdots,x^{*}_{n},z^{*})$ is a feasible Nash point of $F$ defined in Problem 1. That is

[TABLE]

The proof of Theorem 3.2 is detailed in Section 0.C in the supplementary materials 1. The third theorem states that our proposed ADMM can achieve a sublinear convergence rate of $o(1/k)$ under Assumption 1, despite the nonconvex and complex nature of Problem 1. Such a rate is state-of-the-art even compared to those methods for simpler convex problems. The theorem is shown as follows:

Theorem 3.3 (Convergence Rate)

If $\rho>2H$, for a bounded sequence $(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$, define $u_{k}=\min\nolimits_{0\leq l\leq k}(\|z^{l+1}-z^{l}\|^{2}_{2}+\sum\nolimits_{i=1}^{n}\|A_{i}(x_{i}^{l+1}-x_{i}^{l})\|^{2}_{2})$, then the convergence rate of $u_{k}$ is $o(1/k)$.

The proof of this theorem is in Section 0.C in the supplementary materials 1. The $o(1/k)$ convergence rate of the proposed ADMM is consistent with much existing work analyzing the convex ADMM, including [12, 17, 22]. Our contribution in term of convergence rate is that we extend the guarantee of $o(1/k)$ into the multi-convex Problem 1.

Our proposed ADMM is more general than some influential works in terms of formulation. The relations between our proposed ADMM and previous works are summarized as follows:

1. Generalization of Block Coordinate Descent (BCD) for multi-convex problems. When the linear constraint $\sum\nolimits_{i=1}^{n}A_{i}x_{i}=z$ is removed in Problem 1, then the proposed ADMM is reduced to the Block Coordinate Descent [39].

2. Generalization of multi-block ADMM. When $f(x_{1},\cdots,x_{n})=0$, the proposed ADMM is reduced to the convex multi-block ADMM [27], i.e. the ADMM with no less than three variables.

Apart from general formulations, the convergence guarantees of our proposed ADMM cover more applications than previous literature. For example, [37] requires the coupled objective $f(x_{1},\cdots,x_{n})$ to be Lipschitz differentiable. However, some important applications such as weakly-constrained multi-task learning (Section 4.1) and learning with signed-network constraints (Section 4.2) do not satisfy this condition. But they are covered by our convergence guarantees of the multi-convex ADMM to a Nash point.

4 Applications

In this section, we apply our proposed ADMM to two real-world applications, both of which conform to Problem 1 and benefit from the convergence properties of the proposed ADMM.

4.1 Weakly-constrained Multi-task Learning

In multi-task learning problems, multiple tasks are learned jointly to achieve a better performance compared with learning tasks independently [32]. Most work on multi-task learning has tended to enforce the assumption of similarity among the feature weight values across tasks [2, 11, 36, 32, 43] because this makes it possible to use convex regularization terms like $\ell_{2,1}$ norms [36] and Graph Laplacians [43]. However, this assumption is usually too strong and is seldom satisfied by the real-world data. Instead of requiring feature weights to be similar in magnitude, a more conservative but probably more reasonable assumption is that multiple tasks share similar polarities for the same feature, which means that if a feature is positively relevant to the output of a task, then its weight will also be positive for other related tasks. This assumption is appropriate for many applications. For example, the feature ‘number of clinic visits’ will be positively related to flu outbreaks, while the feature ‘popularity of vaccination’ will be negatively related to them, even though their feature weights can vary dramatically for different countries (namely tasks here). This is achieved by enforcing the requirement for every pair of tasks with neighboring indices to have the same weight sign. This optimization objective is shown as follows:

[TABLE]

where $n$ and $m$ denote the number of tasks and features, respectively, $w_{i,j}$ is the weight of the $j$-th feature in the $i$-th task, $w_{i}$ is the weight of the $i$-th task, and $Loss_{i}(w_{i})$ and $\Omega_{i}(w_{i})$ are the loss function and the regularization term of the $i$-th task, respectively. The inequality constraint implies that the $i$-th and the $i+1$-th tasks share the same sign for their weights. Equation (5) is rewritten in the following form to fit in our proposed ADMM framework:

[TABLE]

where $z=[z_{1};\cdots;z_{n}]$ is an auxiliary variable, and $\lambda_{1}>0$ is a tuning parameter. Notice that the inequality constraint $w_{i,j}w_{i+1,y}\geq 0$ is transformed to a quadratic penalty $c_{1}(x)$ such that $c_{1}(x)=\begin{cases}x^{2}&x<0\\ 0&x\geq 0\end{cases}$ which makes the formulation consistent with Problem 1. The proposed ADMM algorithm for this case is shown in Appendix 0.D.1 in the supplementary materials 1.

4.2 Learning with Signed-Network Constraints

The application of network models for social network analysis has attracted the attention of a large number of researchers [7]. For example, influential societal events often spread across many social networking sites and are expressed in different languages. Such multi-lingual indicators usually transmit similar semantic information through networks and have thus been utilized to facilitate social event forecasting [41]. The problem with network constraints is formulated as follows:

[TABLE]

where $\beta_{i}$ is the weight of the $i$-th node. $Loss(\beta_{1},\cdots,\beta_{n})$ is a loss function and $\omega_{i}(\beta_{i})$ is a regularization term for the $i$-th node. $E_{s}=\{(\beta_{i},\beta_{j})|\beta_{i}\beta_{j}\geq 0\}$ and $E_{d}=\{(\beta_{p},\beta_{q})|\beta_{p}\beta_{q}\leq 0\}$ are two edge sets to represent two opposite relationships: $(\beta_{i},\beta_{j})\in E_{s}$ means that $\beta_{i}\beta_{j}\geq 0$, while $(\beta_{p},\beta_{q})\in E_{d}$ means that $\beta_{p}\beta_{q}\leq 0$. The constraint means that some pair $(\beta_{i},\beta_{j})$ satisfies the edge set $E_{s}$, and that some pair $(\beta_{p},\beta_{q})$ satisfies the edge set $E_{d}$. For example, in the problem of social event forecasting with French and English, $E_{s}$ and $E_{d}$ are edge sets of synonyms and antonyms between French and English, and the weight pair of the French word ”bien” and the English word ”good” belongs to $E_{s}$. The problem with network constraints can be reformulated approximately to the following:

[TABLE]

where $z=[z_{1};\cdots;z_{n}]$ is an auxiliary variable, and $\lambda_{2}>0$ is a tuning parameter. The constraint $(\beta_{i},\beta_{j})\in E_{s}$ and $(\beta_{p},\beta_{q})\in E_{d}(1\leq i,j,p,q\leq n)$ are transformed to two quadratic penalties $c_{2}(\beta_{i},\beta_{j})$ and $c_{3}(\beta_{p},\beta_{q})$ as follows:

$c_{2}(\beta_{i},\beta_{j})=\begin{cases}(\beta_{i}\beta_{j})^{2}&(\beta_{i},\beta_{j})\not\in E_{s}\\ 0&(\beta_{i},\beta_{j})\in E_{s}\end{cases},c_{3}(\beta_{p},\beta_{q})=\begin{cases}(\beta_{p}\beta_{q})^{2}&(\beta_{p},\beta_{q})\not\in E_{d}\\ 0&(\beta_{p},\beta_{q})\in E_{d}\end{cases}$.

The proposed ADMM for this case is also shown in Appendix 0.D.2 in the supplementary materials 1.

5 Experiments

In this section, we test our proposed ADMM using ten real-world datasets on two applications detailed in Section 4. Scalability, effectiveness, and convergence properties are compared with several existing state-of-the-art methods on ten real datasets. All experiments were conducted on a 64-bit machine with Intel(R) Core(TM) processor (i7-6820HQ CPU@ 2.70GHZ) and 16.0GB memory.

5.1 Experiment I: Weak-constrained Multi-task Learning

To evaluate the effectiveness of our method on the application of weak-constrained multi-task learning described in Equation (6), a real-world school dataset is used to evaluate the effectiveness of our proposed ADMM. It consists of the examination scores in three years of 15,362 students from 139 secondary schools, which are treated as tasks for examination scores prediction based on 27 input features such as year of the examination, school-specific features, and student-specific features. The dataset is publicly available and the detailed description can be found in the original paper [21]. $\rho$ was set to $1000$. Here we chose two kinds of $\lambda_{1}$: (1) $\lambda_{1}^{k}=10^{5}$; (2) $\lambda^{k+1}_{1}=\lambda^{k}_{1}+10$ with $\lambda_{1}^{k}=1$. $\lambda_{1}(1)$ and $\lambda_{1}(2)$ are the first and the second choice of $\lambda_{1}$, respectively.

Metrics. In this experiment, five metrics were utilized to evaluate model performance. Mean Squared Error (MSE) measures the average of the squares of the difference between observation and estimation. Different from MSE, Mean Squared Logarithmic Error (MSLE) measures the ratio of observation to estimation. Mean Absolute Error (MAE) is also an error measurement but computed in the absolute value. The less the above three metrics are, the better a regression model is. Explained Variance (EV) computes the ratio of the variance of the error to that of observation. The coefficient of determination or R2 score is the proportion of the variance in the dependent variable that is predictable from the independent variable. The higher score of EV and R2 are, the better a regression model is.

Baselines. To validate the effectiveness of the proposed ADMM, five benchmark multi-task learning models served as comparison methods. Loss functions were set to least square errors. The number of iterations was set to $5,000$. The regularization parameter $\alpha$ was set based on 5-fold cross-validation on the training set.

1. multi-task learning with Joint Feature Selection (JFS) [2, 43] . JFS is one of the most commonly used strategies in multi-task learning. It captures the relatedness of multiple tasks by a constraint of a weight matrix to share a common set of features. $\alpha$ was set to 100.

2. Clustered Multi-Task Learning (CMTL) [42, 43]. CMTL assumes that multiple tasks are clustered into several groups. Tasks in the same group are similar to each other. $\alpha$ was set to 1.

3. multi-task Lasso (mtLasso) [43]. mtLasso extends the classic Lasso model to the multi-task learning setting. $\alpha$ was set to 10.

4. a convex relaxation of Alternating Structure Optimization (cASO) [43, 1]. cASO decomposes each task into two components: task-specific feature mapping and task-shared feature mapping. $\alpha$ was set to 0.01.

5. Block Coordinate Descent (BCD) [39]. BCD is an intuitive method to solve multi-convex problems, which optimizes each variable alternately. $\alpha$ was set to 10.

Performance. As discussed in Section 4.1, the convergence of our proposed ADMM is guaranteed based on our theoretical framework. To verify this, Figures 1(a) and 1(b) illustrate the residual and objective values in different iterations, which demonstrates the convergence of the proposed ADMM on this nonconvex problem. Then the performance of examination score prediction on this dataset is illustrated in Table 1. Table 1 shows the mean and the standard deviation of all methods, which were repeated 10 times by initializing parameters randomly, to make experimental evaluation robust. It shows that $\lambda_{1}(2)$ outperforms $\lambda_{1}(1)$ in four out of five metrics for the proposed ADMM. In addition, the proposed ADMM achieves the best performance in all the metrics, compared to all comparison methods. Moreover, the standard deviation of the proposed ADMM is about $30\%$ smaller than any other comparison method. This is because our method only enforces that the sign of the feature weight across different tasks is the same, while comparison methods typically perform too aggressive assumptions on the similarity among tasks. For example, CMTL enforces that the correlated tasks need to have similar feature weights using squared regularization on the difference between feature weights. JFS and mtLasso still tend to enforce similar weights on features in different tasks by $\ell_{2,1}$ norm. Because their enforcement is weaker than CMTL, their performance is better. cASO gets relatively weak performance because it optimizes an approximation of a nonconvex problem, and thus the approximate solution may be distant from the true solution to the original problem. Finally, the BCD performs the worst among all methods, even though it shares the same formulation with our proposed ADMM. This reflects the advantage of our proposed ADMM algorithm: dual information in one iteration can be passed to its following iteration by dual variables, which yields better performance.

Scalability. To investigate the scalability of the proposed ADMM compared with all baselines in Experiment I, we measured the training time of them in the school dataset when the number of features varies. The training time was averaged by running 20 times. Figure 2 shows the training time of all methods when the number of features ranges from 10 to 28. The training time of all methods increased linearly concerning the number of features. cASO was the most efficient of all methods, while the proposed ADMM was ranked second. mtLasso and JFS also trained a model within 5 seconds on average. CMTL was time-consuming for training, which spent more than 10 seconds.

5.2 Experiment II: Event Forecasting with Multi-lingual Indicators

Datasets. To evaluate the performance of our proposed ADMM on the application in Section 4.2, extensive experiments on nine real-world datasets have been performed. The dataset is obtained by randomly sampling 10% (by volume) of the Twitter data from Jan 2013 to Dec 2014. The data in the first and second years are used and training and test set, respectively. For the topic (i.e., social unrest) of interest, 1,806 keywords in the three major languages in Latin America, namely English, Spanish, and Portuguese, were provided by the paper [41]. Their translation relationships have also been labeled as semantic links among them, such as “protest” in English, “protesta” in Spanish, and “protesto” in Portuguese. The event forecasting results were validated against a labeled event set, known as the Gold Standard Report (GSR), which is publicly available [25].

Metric and Baselines. The metric used to evaluate the performance is Area Under the receiver operating characteristic Curve (AUC). Five comparison methods including the state-of-the-art Multi-Task learning (MTL), Multi-Resolution Event Forecasting (MREF), and Distant-supervision of Heterogeneous Multitask Learning (DHML) as well as classic methods Logistic Regression (LogReg) and Lasso. $\rho$ was set to $10$. Here we chose two kinds of $\lambda_{2}$: (1) $\lambda_{2}^{k}=10^{5}$; (2) $\lambda^{k+1}_{2}=\lambda^{k}_{2}+10$ with $\lambda_{2}^{k}=1$. $\lambda_{2}(1)$ and $\lambda_{2}(2)$ are the first and the second choice of $\lambda_{2}$, respectively. All the hyper-parameters were tuned by 5-fold cross-validation.

Performance. As shown in Table 2, $\lambda_{2}(2)$ outperforms $\lambda_{2}(1)$ marginally in seven out of nine datasets for the proposed ADMM, and they generally perform the best among all the methods, with DHML and BCD the second-best performer. They all outperform the others typically by at least 5%-10%. This is because they leverage the multilingual correlation among the features to boost up the model’s generalizability. Thanks to the framework of multi-task learning, MTL and MREF obtained a competitive performance with AUC typically over 0.7, which outperforms simple methods like LogReg and LASSO by 5% on average.

Efficiency. In Experiment II, we also compared the training time of the proposed ADMM in comparison with all baselines on 9 datasets. The training time was averaged by running 5 times. The training time was shown in Table 3. We do not show BCD because its training time is similar to the proposed ADMM. Overall, the proposed ADMM was the most efficient of all methods for all datasets. It consumed no more than 30 seconds on all datasets. MTL ranked second, but it spent hundreds of seconds on some datasets, like BR and MX. As the most time-consuming baselines, LogReg and MREF trained a model in thousands of seconds or more.

6 Conclusions

We propose an ADMM framework for multi-convex problems with multiple coupled variables. It not only inherits the merits of general ADMMs but also provides advantageous theoretical properties on convergence conditions and properties under mild conditions. Besides, several machine learning applications of recent interest are discussed as special cases of our proposed ADMM. Extensive experiments have been conducted on ten real-world datasets, and demonstrate the effectiveness, scalability, and convergence properties of our proposed ADMM.

Acknowledgement

This work was supported by the National Science Foundation (NSF) Grant No. 1755850, No. 1841520, No. 2007716, No. 2007976, No. 1942594, No. 1907805, a Jeffress Memorial Trust Award, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcontract No: 10827.002.120.04).

Appendix 0.A Preliminary Lemmas

In this section, we give preliminary lemmas which are also used in the proof the proofs of Lemmas 1 and 2. While Lemmas 4 and 5 depend on the optimality conditions of subproblems, Lemmas 3, 6 and 7 require Assumption 1.

Lemma 3

It holds that $\forall z_{1},z_{2}\in R^{q}$,

[TABLE]

Proof

Because $h(z)$ is Lipschitz differentiable by Assumption 1, so is $-h(z)$. Therefore, this lemma is proven exactly as same as Lemma 2.1 in [4].

Lemma 4

It holds that $y^{k}=\nabla h(z^{k})$ for all $k\in\mathbb{N}$.

Proof

The optimality condition for the problem with regard to $z^{k}$ gives rise to

[TABLE]

Because $y^{k}=y^{k-1}+\rho(\sum\nolimits A_{i}x_{i}^{k}-z^{k})$, we have $y^{k}=\nabla h(z^{k})$.

Lemma 5

It holds that for $\forall k\in\mathbb{N}$,

[TABLE]

Proof

[TABLE]

where the second equality follows from the cosine rule: $\|b+c\|^{2}-\|a+c\|^{2}=\|b-a\|^{2}+2(a+c)^{T}(b-a)$ with $a=A_{i}x_{i}^{k+1}$, $b=A_{i}x_{i}^{k}$ and $c=\sum\nolimits_{j=1}^{i-1}A_{j}x_{j}^{k+1}+\sum\nolimits_{j=i+1}^{n}A_{j}x_{j}^{k}-z^{k}$.

The optimality condition of $x^{k+1}_{i}$ leads to

[TABLE]

We have the following result according to the definition of subgradient

[TABLE]

Therefore, the lemma is proved.

Lemma 6

If $\rho>2H$ so that $C_{1}=\rho/2-H/2-H^{2}/\rho>0$, then it holds that

[TABLE]

Proof

[TABLE]

We choose $\rho>2H$ to make $C_{1}>0$.

Lemma 7

$\forall k\in\mathbb{N}$, we have $\|y^{k+1}-y^{k}\|\leq H\|z^{k+1}-z^{k}\|$.

Proof

[TABLE]

Appendix 0.B Proofs of Lemmas 1- 2

Proof (Proof of Lemma 1)

This follows directly from Lemmas 5 and 6.

Proof (Proof of Lemma 2)

There exists $z^{\prime}$ such that $\sum\nolimits_{i=1}^{n}A_{i}x_{i}^{k}-z^{\prime}=0$. Therefore, we have

[TABLE]

where $S=\{f(x_{1},\cdots,x_{n})+h(z):\sum\nolimits_{i=1}^{n}A_{i}x_{i}-z=0\}$, which is the objective value of Problem 1, and therefore bounded from below. Then we have

[TABLE]

Therefore, $L_{\rho}(x_{1}^{k},\cdots,x_{n}^{k},z^{k},y^{k})$ is bounded from below.

Appendix 0.C Proofs of Theorems 3.1-3.3

Proof (Proof of Theorem 3.1)

We show residual convergence and objective convergence based on Lemmas 1 and 2.

From Lemma 1, $L_{\rho}(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$ decreases monotonically, and $L_{\rho}(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$ is lower bounded by Lemma 2. Therefore, $L_{\rho}(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$ is convergent because a monotone bounded sequence converges (Monotone Convergence Theorem). According to the continuity of $L_{\rho}$, we take $k\rightarrow\infty$ on the both sides of Inequality (4) to obtain

[TABLE]

On one hand, $L_{\rho}(x_{1},\cdots,x_{n},z,y)$ is convergent, so we have

[TABLE]

On the other hand, $C_{2}(\|z^{k+1}-z^{k}\|^{2}_{2}+\sum\nolimits_{i=1}^{n}\|A_{i}(x_{i}^{k+1}-x_{i}^{k})\|^{2}_{2}$ is nonnegative, so we get

[TABLE]

This suggests that $\lim_{k\rightarrow\infty}(z^{k+1}-z^{k})=0$ and $\lim_{k\rightarrow\infty}A_{i}(x_{i}^{k+1}-x_{i}^{k})=0(i=1,\cdots,n)$. Moreover, by Lemma 7, $\lim_{k\rightarrow\infty}\|y^{k+1}-y^{k}\|\leq H\lim_{k\rightarrow\infty}\|z^{k+1}-z^{k}\|=0$. So we have $\lim_{k\rightarrow\infty}(y^{k+1}-y^{k})=0$.

a). For residual convergence, by the Line 8 of Algorithm 1, we have

[TABLE]

b). For objective convergence, since

[TABLE]

and $L_{\rho}(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$ is convergent, $r^{k}$ converges to 0 and $y^{k}$ is bounded, then $F(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$ is also convergent.

Proof (Proof of Theorem 3.2)

Obviously $\lim_{k\rightarrow\infty}(z^{k+1}-z^{k})=0$ and $\lim_{k\rightarrow\infty}(y^{k+1}-y^{k})=0$ from the proof of Theorem 3.1. In order to prove this theorem, we firstly prove that $\lim_{k\rightarrow\infty}(x^{k+1}_{i}-x^{k}_{i})=0(i=1,\cdots,n)$ if either of two assumptions holds, then prove that any limit point $(x^{*}_{1},\cdots,x^{*}_{n},z^{*})$ is a feasible Nash point of Problem 1.

(a). Suppose $A_{i}(i=1,\cdots,n)$ have full rank. Because $\lim_{k\rightarrow\infty}A_{i}(x^{k+1}_{i}-x^{k}_{i})=0$ from the proof of Theorem 3.1, then obviously $\lim_{k\rightarrow\infty}(x^{k+1}_{i}-x^{k}_{i})=0$ [22].

(b). Suppose $F$ is strongly convex with regard to $x_{i}$. Because $L_{\rho}(x_{1},\cdots,x_{n},z,y)=F(x_{1},\cdots,x_{n},z)+y^{T}(\sum\nolimits A_{i}x_{i}-z)+(\rho/2)\|\sum\nolimits A_{i}x_{i}-z\|^{2}_{2}$, $F(x_{1},\cdots,x_{n},z)$, and $y^{T}(\sum\nolimits A_{i}x_{i}-z)+(\rho/2)\|\sum\nolimits A_{i}x_{i}-z\|^{2}_{2}$ are strongly convex, $L_{\rho}$ is also strongly convex regard to $x_{i}$ [23] with the assumed constant $D_{i}>0$. We have

[TABLE]

where $\forall v_{i}^{k+1}\in\partial_{x_{i}}L_{\rho}(x^{k+1}_{1},\cdots,x^{k+1}_{i-1},x^{k+1}_{i},x^{k}_{i+1},\cdots,x^{k}_{n},z^{k},y^{k})$. The optimality condition of $x^{k+1}_{i}$ leads to

$0\in\partial_{x_{i}}L_{\rho}(x^{k+1}_{1},\cdots,x^{k+1}_{i-1},x^{k+1}_{i},x^{k}_{i+1},\cdots,x^{k}_{n},z^{k},y^{k})$. Therefore, we have

[TABLE]

We sum up Inequality (10) from $i=1,\cdots,n$ and Inequality (9) to obtain

[TABLE]

where $C_{1}>0$ by Lemma 6 if $\rho>2H$. According to the continuity of $L_{\rho}$, we take $k\rightarrow\infty$ on the both sides of Inequality (11) to obtain

[TABLE]

On one hand, $L_{\rho}(x_{1},\cdots,x_{n},z,y)$ is convergent, so we have

[TABLE]

On the other hand, $\sum\nolimits_{i=1}^{n}(D_{i}/2)\|x_{i}^{k+1}-x_{i}^{k}\|^{2}_{2}+C_{1}\|z^{k+1}-z^{k}\|^{2}_{2}$ is nonnegative, so we get

[TABLE]

This suggests that $\lim_{k\rightarrow\infty}(x_{i}^{k+1}-x_{i}^{k})=0(i=1,\cdots,n)$ and $\lim_{k\rightarrow\infty}(z^{k+1}-z^{k})=0$.

Therefore, $\lim_{k\rightarrow\infty}(x_{i}^{k+1}-x_{i}^{k})=0(i=1,\cdots,n)$ if either of two assumptions holds. Because $(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$ is bounded, there exists a subsequence $(x_{1}^{s},\cdots,x_{n}^{s},z^{s},y^{s})$ such that $(x_{1}^{s},\cdots,x_{n}^{s},z^{s},y^{s})\rightarrow(x_{1}^{*},\cdots,x_{n}^{*},z^{*},y^{*})$ where $(x_{1}^{*},\cdots,x_{n}^{*},z^{*},y^{*})$ is a limit point. Because $\lim_{s\rightarrow\infty}(x_{i}^{s+1}-x_{i}^{s})=0(i=1,\cdots,n)$, $\lim_{s\rightarrow\infty}(z^{s+1}-z^{s})=0$ and $\lim_{s\rightarrow\infty}(y^{s+1}-y^{s})=0$, we have $(x_{1}^{s+1},\cdots,x_{n}^{s+1},z^{s+1},y^{s+1})\rightarrow(x_{1}^{*},\cdots,x_{n}^{*},z^{*},y^{*})$. Now we prove that the limit point $(x_{1}^{*},\cdots,x_{n}^{*},z^{*})$ is a feasible Nash point of Problem 1.

For feasibility, since $\lim_{k\rightarrow\infty}r^{k}=\lim_{k\rightarrow\infty}\sum\nolimits_{i=1}^{n}A_{i}x^{k}_{i}-z^{k}=0$, so for the subsequence $(x_{1}^{s},\cdots,x_{n}^{s},z^{s},y^{s})\rightarrow(x_{1}^{*},\cdots,x_{n}^{*},z^{*},y^{*})$, we have $\lim_{s\rightarrow\infty}r^{s}=\lim_{s\rightarrow\infty}(\sum\nolimits_{i=1}^{n}A_{i}x^{s}_{i}-z^{s})=0$ then $\sum\nolimits_{i=1}^{n}A_{i}x^{*}_{i}-z^{*}=0$.

For the Nash point, we obtain the following according to the optimality conditions of $x^{s+1}_{i}(i=1,\cdots,n)$ and $z^{s+1}$ in Equations (2) and (3), respectively.

[TABLE]

According to the continuity of $L_{\rho}$, we take $s\rightarrow\infty$ on the both sides of two inequalities. Because $(x_{1}^{s},\cdots,x_{n}^{s},z^{s},y^{s})\rightarrow(x_{1}^{*},\cdots,x_{n}^{*},z^{*},y^{*})$ and $(x_{1}^{s+1},\cdots,x_{n}^{s+1},z^{s+1},y^{s+1})\rightarrow(x_{1}^{*},\cdots,x_{n}^{*},z^{*},y^{*})$, we have

[TABLE]

Here $\forall(x^{*}_{1},\cdots,x^{*}_{i-1},x_{i},x^{*}_{i+1},\cdots,x^{*}_{n},z^{*})\in dom(F)$ and $\forall(x^{*}_{1},\cdots,x^{*}_{n},z)\in dom(F)$ mean $\forall x_{i}\ s.t.\ \sum\nolimits_{j=1,j\neq i}^{n}A_{j}x^{*}_{j}+A_{i}x_{i}-z^{*}=0$ and $\forall z\ s.t.\ \sum\nolimits_{j=1}^{n}A_{j}x^{*}_{j}-z=0$, respectively. Using the fact that $(x^{*}_{1},\cdots,x^{*}_{n},z^{*})$ is feasible in Problem 1, we obtain $L_{\rho}(x^{*}_{1},\cdots,z^{*},y^{*})=F(x^{*}_{1},\cdots,z^{*})$, $L_{\rho}(x^{*}_{1},\cdots,x^{*}_{i-1},x_{i},x^{*}_{i+1},\cdots,x^{*}_{n},z^{*},y^{*})=F(x^{*}_{1},\cdots,x^{*}_{i-1},x_{i},x^{*}_{i+1},\cdots,x^{*}_{n},z^{*})$ and $L_{\rho}(x^{*}_{1},\cdots,x^{*}_{n},z,y^{*})=F(x^{*}_{1},\cdots,x^{*}_{n},z)$. Therefore, we prove that $(x^{*}_{1},\cdots,x^{*}_{n},z^{*})$ is a feasible Nash point of $F$ defined in Problem 1.

Proof (Proof of Theorem 3.3)

To prove this theorem, we will first show that $u_{k}$ satisfies two conditions: (1). $u_{k}\geq u_{k+1}$. (2). $\sum\nolimits_{k=0}^{\infty}u_{k}$ is bounded. We then conclude the convergence rate of $o(1/k)$ based on these two conditions. Specifically, first, we have

[TABLE]

Therefore $u_{k}$ satisfies the first condition. Second,

[TABLE]

where $L^{*}_{\rho}=\lim_{k\rightarrow\infty}L_{\rho}(x^{k}_{1},\cdots,x^{k}_{n},z^{k},y^{k})$. So $\sum\nolimits_{k=0}^{\infty}u_{k}$ is bounded and $u_{k}$ satisfies the second condition. Finally, it has been proved that the sufficient conditions of convergence rate $o(1/k)$ are: (1) $u_{k}\geq u_{k+1}$, and (2) $\sum\nolimits_{k=0}^{\infty}u_{k}$ is bounded, and (3) $u_{k}\geq 0$ (Lemma 1.2 in [12]). Since we have proved the first two conditions and the third one $u_{k}\geq 0$ is obvious, the convergence rate of $o(1/k)$ is proven.

Appendix 0.D Algorithms for Applications

0.D.1 Weakly-constrained Multi-task Learning

Applying the proposed ADMM to solve the problem in Equation (6), we get Algorithm 2. Specifically, Lines 4-9 update primal variables $w_{i}(i=1,\cdots,n)$ and $z_{i}(i=1,\cdots,n)$, Line 10 updates the dual variable $y_{i}(i=1,\cdots,n)$.

All subproblems are detailed as follows:

1. Update $w^{k+1}$

The $w^{k+1}_{i}(i=1,\cdots,n)$ are updated as follows:

[TABLE]

They can be solved by the Iterative Soft Thresholding Algorithm (ISTA) [4]. Take $w_{1}^{k+1}$ as an example, The ISTA leads to

[TABLE]

where $w_{1}^{t}$ is the $t$-th iteration in the ISTA, $\eta>0$ is a learning rate, $\phi(w^{t}_{1})=Loss_{1}(w^{t}_{1})+\rho/2\|w^{t}_{1}-z^{k}_{1}+y^{k}_{1}/\rho\|^{2}_{2}$. For each entry of $w^{t+1}_{1,j}(j=1,2,\cdots,m)$, we have the following closed-form solution as follows:

1). If $w^{t+1}_{1,j}w^{k}_{2,j}\leq 0$, then $w^{t+1}_{1,j}\leftarrow(w_{1,j}^{t}-\eta\nabla_{j}\phi(w_{1}^{t}))/(2\eta\lambda_{1}w^{2}_{2,j}+1)$.

2). If $w^{t+1}_{1,j}w^{k}_{2,j}\geq 0$, then $w_{1,j}^{t+1}\leftarrow w_{1,j}^{t}-\eta\nabla_{j}\phi(w_{1}^{t})$. where $\nabla_{j}\phi(w_{1}^{t})$ is the $j$-th entry of $\nabla\phi(w_{1}^{t})$.

2. Update $z^{k+1}$

The $z^{k+1}_{i}(i=1,\cdots,n)$ are updated as follows:

[TABLE]

For $\ell_{1}$ or $\ell_{2}$ regularization, they have closed-form solutions.

0.D.2 Learning with Signed-Network Constraints

Applying proposed ADMM to solve the problem in Equation (7), we get Algorithm 3. Specifically, Lines 4-7 update primal variables $\beta_{i}(i=1,\cdots,n)$ and $z$, Line 8 updates the dual variable $y$.

All subproblems are detailed as follows:

1. Update $\beta^{k+1}$

The $\beta^{k+1}_{i}(i=1,\cdots,n)$ are updated as follows:

[TABLE]

Similar to updating $w^{k+1}_{i}$ in Algorithm 2, they can be solved efficiently by the ISTA [4].

2. Update $z^{k+1}$

The $z^{k+1}_{i}(i=1,\cdots,n)$ are updated as follows:

[TABLE]

Similar to updating $z^{k+1}_{i}$ in Algorithm 2, they usually have closed-form solutions.