TL;DR
This paper introduces dlADMM, a novel ADMM-based optimization framework for deep learning that guarantees global convergence, improves efficiency, and outperforms existing methods on benchmark datasets.
Contribution
The paper presents the first global convergence proof for ADMM in deep learning and reduces computational complexity from cubic to quadratic in feature dimensions.
Findings
dlADMM outperforms most comparison methods on benchmarks.
Reduces time complexity from cubic to quadratic.
Provides the first global convergence proof for ADMM in deep neural networks.
Abstract
Alternating Direction Method of Multipliers (ADMM) has been used successfully in many conventional machine learning applications and is considered to be a useful alternative to Stochastic Gradient Descent (SGD) as a deep learning optimizer. However, as an emerging domain, several challenges remain, including 1) The lack of global convergence guarantees, 2) Slow convergence towards solutions, and 3) Cubic time complexity with regard to feature dimensions. In this paper, we propose a novel optimization framework for deep learning via ADMM (dlADMM) to address these challenges simultaneously. The parameters in each layer are updated backward and then forward so that the parameter information in each layer is exchanged efficiently. The time complexity is reduced from cubic to quadratic in (latent) feature dimensions via a dedicated algorithm design for subproblems that enhances them…
| Notations | Descriptions |
|---|---|
| Number of layers. | |
| The weight matrix for the -th layer. | |
| The intercept vector for the -th layer. | |
| The temporary variable of the linear mapping for the -th layer. | |
| The nonlinear activation function for the -th layer. | |
| The output for the -th layer. | |
| The input matrix of the neural network. | |
| The predefined label vector. | |
| The risk function for the -th layer. | |
| The regularization term for the -th layer. | |
| The number of neurons for the -th layer. |
| MNIST dataset: From 200 to 1,000 neurons | |||||
|---|---|---|---|---|---|
| 200 | 400 | 600 | 800 | 1000 | |
| 1.9025 | 2.7750 | 3.6615 | 4.5709 | 5.7988 | |
| 2.8778 | 4.6197 | 6.3620 | 8.2563 | 10.0323 | |
| 2.2761 | 3.9745 | 5.8645 | 7.6656 | 9.9221 | |
| 2.4361 | 4.3284 | 6.5651 | 8.7357 | 11.3736 | |
| 2.7912 | 5.1383 | 7.8249 | 10.0300 | 13.4485 | |
| Fashion MNIST dataset: From 200 to 1,000 neurons | |||||
| 200 | 400 | 600 | 800 | 1000 | |
| 2.0069 | 2.8694 | 4.0506 | 5.1438 | 6.7406 | |
| 3.3445 | 5.4190 | 7.3785 | 9.0813 | 11.0531 | |
| 2.4974 | 4.3729 | 6.4257 | 8.3520 | 10.0728 | |
| 2.7108 | 4.7236 | 7.1507 | 9.4534 | 12.3326 | |
| 2.9577 | 5.4173 | 8.2518 | 10.0945 | 14.3465 | |
| MNIST dataset: From 11,000 to 55,000 training samples | |||||
|---|---|---|---|---|---|
| 11,000 | 22,000 | 33,000 | 44,000 | 55,000 | |
| 1.0670 | 2.0682 | 3.3089 | 4.6546 | 5.7709 | |
| 2.3981 | 3.9086 | 6.2175 | 7.9188 | 10.2741 | |
| 2.1290 | 3.7891 | 5.6843 | 7.7625 | 9.8843 | |
| 2.1295 | 4.1939 | 6.5039 | 8.8835 | 11.3368 | |
| 2.5154 | 4.9638 | 7.6606 | 10.4580 | 13.4021 | |
| Fashion MNIST dataset: From 12,000 to 60,000 training samples | |||||
| 12,000 | 24,000 | 36,000 | 48,000 | 60,000 | |
| 1.2163 | 2.3376 | 3.7053 | 5.1491 | 6.7298 | |
| 2.5772 | 4.3417 | 6.6681 | 8.3763 | 11.0292 | |
| 2.3216 | 4.1163 | 6.2355 | 8.3819 | 10.7120 | |
| 2.3149 | 4.5250 | 6.9834 | 9.5853 | 12.3232 | |
| 2.7381 | 5.3373 | 8.1585 | 11.1992 | 14.2487 | |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsAlternating Direction Method of Multipliers
ADMM for Efficient Deep Learning with Global Convergence
Junxiang Wang, Fuxun Yu, Xiang Chen and Liang Zhao
George Mason University
jwang40,fyu2,xchen26,[email protected]
(2019)
Abstract.
Alternating Direction Method of Multipliers (ADMM) has been used successfully in many conventional machine learning applications and is considered to be a useful alternative to Stochastic Gradient Descent (SGD) as a deep learning optimizer. However, as an emerging domain, several challenges remain, including 1) The lack of global convergence guarantees, 2) Slow convergence towards solutions, and 3) Cubic time complexity with regard to feature dimensions. In this paper, we propose a novel optimization framework for deep learning via ADMM (dlADMM) to address these challenges simultaneously. The parameters in each layer are updated backward and then forward so that the parameter information in each layer is exchanged efficiently. The time complexity is reduced from cubic to quadratic in (latent) feature dimensions via a dedicated algorithm design for subproblems that enhances them utilizing iterative quadratic approximations and backtracking. Finally, we provide the first proof of global convergence for an ADMM-based method (dlADMM) in a deep neural network problem under mild conditions. Experiments on benchmark datasets demonstrated that our proposed dlADMM algorithm outperforms most of the comparison methods.
Deep Learning, Global Convergence, Alternating Direction Method of Multipliers
††journalyear: 2019††copyright: acmcopyright††conference: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 4–8, 2019; Anchorage, AK, USA††booktitle: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’19), August 4–8, 2019, Anchorage, AK, USA††price: 15.00††doi: 10.1145/3292500.3330936††isbn: 978-1-4503-6201-6/19/08††ccs: Theory of computation Nonconvex optimization††ccs: Computing methodologies Neural networks
1. Introduction
Deep learning has been a hot topic in the machine learning community for the last decade. While conventional machine learning techniques have limited capacity to process natural data in their raw form, deep learning methods are composed of non-linear modules that can learn multiple levels of representation automatically (lecun2015deep, ). Since deep learning methods are usually applied in large-scale datasets, such approaches require efficient optimizers to obtain a feasible solution within realistic time limits.
Stochastic Gradient Descent (SGD) and many of its variants are popular state-of-the-art methods for training deep learning models due to their efficiency. However, SGD suffers from many limitations that prevent its more widespread use: for example, the error signal diminishes as the gradient is backpropagated (i.e. the gradient vanishes); and SGD is sensitive to poor conditioning, which means a small input can change the gradient dramatically. Recently, the use of the Alternating Direction Method of Multipliers (ADMM) has been proposed as an alternative to SGD. ADMM splits a problem into many subproblems and coordinates them globally to obtain the solution. It has been demonstrated successfully for many machine learning applications (boyd2011distributed, ). The advantages of ADMM are numerous: it exhibits linear scaling as data is processed in parallel across cores; it does not require gradient steps and hence avoids gradient vanishing problems; it is also immune to poor conditioning (taylor2016training, ).
Even though the performance of the ADMM seems promising, there are still several challenges at must be overcome: 1. The lack of global convergence guarantees. Despite the fact that many empirical experiments have shown that ADMM converges in deep learning applications, the underlying theory governing this convergence behavior remains mysterious. This is because a typical deep learning problem consists of a combination of linear and nonlinear mappings, causing optimization problems to be highly nonconvex. This means that traditional proof techniques cannot be directly applied. 2. Slow convergence towards solutions. Although ADMM is a powerful optimization framework that can be applied to large-scale deep learning applications, it usually converges slowly to high accuracy, even for simple examples (boyd2011distributed, ). It is often the case that ADMM becomes trapped in a modest solution and hence performs worse than SGD, as the experiment described later in this paper in Section 5 demonstrates. 3. Cubic time complexity with regard to feature dimensions. The implementation of the ADMM is very time-consuming for real-world datasets. Experiments conducted by Taylor et al. found that ADMM required more than 7000 cores to train a neural network with just 300 neurons (taylor2016training, ). This computational bottleneck mainly originates from the matrix inversion required to update the weight parameters. Computing an inverse matrix needs further subiterations, and its time complexity is approximately , where is a feature dimension (boyd2011distributed, ).
In order to deal with these difficulties simultaneously, in this paper we propose a novel optimization framework for a deep learning Alternating Direction Method of Multipliers (dlADMM) algorithm. Specifically, our new dlADMM algorithm updates parameters first in a backward direction and then forwards. This update approach propagates parameter information across the whole network and accelerates the convergence process. It also avoids the operation of matrix inversion using the quadratic approximation and backtracking techniques, reducing the time complexity from to . Finally, to the best of our knowledge, we provide the first proof of the global convergence of the ADMM-based method (dlADMM) in a deep neural network problem. The assumption conditions are mild enough for many common loss functions (e.g. cross-entropy loss and square loss) and activation functions (e.g. rectified linear unit (ReLU) and leaky ReLU) to satisfy. Our proposed framework and convergence proof are highly flexible for fully-connected deep neural networks, as well as being easily extendable to other popular network architectures such as Convolutional Neural Networks (krizhevsky2012imagenet, ) and Recurrent Neural Networks (mikolov2010recurrent, ). Our contributions in this paper include:
- •
We present a novel and efficient dlADMM algorithm to handle the fully-connected deep neural network problem. The new dlADMM updates parameters in a backward-forward fashion to speed up the convergence process.
- •
We propose the use of quadratic approximation and backtracking techniques to avoid the need for matrix inversion as well as reducing the computational cost for large scale datasets. The time complexity of subproblems in dlADMM is reduced from to .
- •
We investigate several attractive convergence properties of dlADMM. The convergence assumptions are very mild to ensure that most deep learning applications satisfy our assumptions. dlADMM is guaranteed to converge to a critical point globally (i.e., whatever the initialization is) when the hyperparameter is sufficiently large. We also analyze the new algorithm’s sublinear convergence rate.
- •
We conduct experiments on several benchmark datasets to validate our proposed dlADMM algorithm. The results show that the proposed dlADMM algorithm performs better than most existing state-of-the-art algorithms, including SGD and its variants.
The rest of this paper is organized as follows. In Section 2, we summarize recent research related to this topic. In Section 3, we present the new dlADMM algorithm, quadratic approximation, and the backtracking techniques utilized. In Section 4, we introduce the main convergence results for the dlADMM algorithm. The results of extensive experiments conducted to show the convergence, efficiency, and effectiveness of our proposed new dlADMM algorithm are presented in in Section 5, and Section 6 concludes this paper by summarizing the research.
2. Related Work
Previous literature related to this research includes optimization for deep learning models and ADMM for nonconvex problems.
Optimization for deep learning models: The SGD algorithm and its variants play a dominant role in the research conducted by deep learning optimization community. The famous back-propagation algorithm was firstly introduced by Rumelhart et al. to train the neural network effectively (rumelhart1986learning, ). Since the superior performance exhibited by AlexNet (krizhevsky2012imagenet, ) in 2012, deep learning has attracted a great deal of researchers’ attention and many new optimizers based on SGD have been proposed to accelerate the convergence process, including the use of Polyak momentum (polyak1964some, ), as well as research on the Nesterov momentum and initialization by Sutskever et al. (sutskever2013importance, ). Adam is the most popular method because it is computationally efficient and requires little tuning (kingma2014adam, ). Other well-known methods that incorporate adaptive learning rates include AdaGrad (duchi2011adaptive, ), RMSProp (tielemandivide, ), and AMSGrad (j.2018on, ). Recently, the Alternating Direction Method of Multipliers (ADMM) has become popular with researchers due to its excellent scalability (taylor2016training, ). However, even though these optimizers perform well in real-world applications, their convergence mechanisms remain mysterious. This is because convergence assumptions are not applicable to deep learning problems, which often require non-differentiable activation functions such as the Rectifier linear unit (ReLU).
ADMM for nonconvex problems: The good performance achieved by ADMM over a range wide of convex problems has attracted the attention of many researchers, who have now begun to investigate the behavior of ADMM on nonconvex problems and made significant advances. For example, Wang et al. proposed an ADMM to solve multi-convex problems with a convergence guarantee (wang2019multi, ), while Wang et al. presented convergence conditions for a coupled objective function that is nonconvex and nonsmooth (wang2015global, ). Chen et al. discussed the use of ADMM to solve problems with quadratic coupling terms (chen2015extended, ) and Wang et al. studied the convergence behavior of the ADMM for problems with nonlinear equality constraints (wang2017nonconvex, ). Even though ADMM has been proposed to solve deep learning applications (taylor2016training, ; gao2019incomplete, ), there remains a lack theoretical convergence analysis for the application of ADMM to such problems.
3. The dlADMM Algorithm
We present our proposed dlADMM algorithm in this section. Section 3.1 formulates the deep neural network problem, Section 3.2 introduces how the dlADMM algorithm works, and the quadratic approximation and backtracking techniques used to solve the subproblems are presented in Section 3.3.
3.1. Problem Formulation
Table 1 lists the important notation utilized in this paper. Even though there are many variants of formulations for deep neural networks, a typical neural network is defined by multiple linear mappings and nonlinear activation functions. A linear mapping for the -th layer is composed of a weight matrix and an intercept vector , where is the number of neurons for the -th layer; a nonlinear mapping for the -th layer is defined by a continuous activation function . Given an input from the -th layer, the -th layer outputs . Obviously, is nested in . By introducing an auxiliary variable , the task of training a deep neural network problem is formulated mathematically as follows:
Problem 1.
[TABLE]
In Problem 1, is the input of the deep neural network where is the number of feature dimensions, and is a predefined label vector. is a risk function for the -th layer, which is convex, continuous and proper, and is a regularization term for the -th layer, which is also convex, continuous, and proper. Rather than solving Problem 1 directly, we can relax Problem 1 by adding an penalty to address Problem 2 as follows:
Problem 2.
[TABLE]
where , , , and is a tuning parameter. Compared with Problem 1, Problem 2 has only a linear constraint and hence is easier to solve. It is straightforward to show that as , the solution of Problem 2 approaches that of Problem 1.
3.2. The dlADMM algorithm
We introduce the dlADMM algorithm to solve Problem 2 in this section. The traditional ADMM strategy for optimizing parameters is to start from the first layer and then update parameters in the following layer sequentially (taylor2016training, ). In this case, the parameters in the final layer are subject to the parameter update in the first layer. However, the parameters in the final layer contain important information that can be transmitted towards the previous layers to speed up convergence. To achieve this, we propose our novel dlADMM framework, as shown in Figure 1. Specifically, the dlADMM algorithm updates parameters in two steps. In the first, the dlADMM begins updating from the -th (final) layer and moves backward toward the first layer. The update order of parameters in the same layer is . In the second, the dlADMM reverses the update direction, beginning at the first layer and moving forward toward the -th (final) layer. The update order of the parameters in the same layer is . The parameter information for all layers can be exchanged completely by adopting this update approach.
Now we can present our dlADMM algorithm mathematically. The augmented Lagrangian function of Problem 2 is shown in the following form:
[TABLE]
where , is a dual variable and is a hyperparameter of the dlADMM algorithm. We denote , , and as the backward update of the dlADMM for the -th layer in the -th iteration , while , , and are denoted as the forward update of the dlADMM for the -th layer in the -th iteration. Moreover, we denote , , ,
, , , , , , , , , , , , and . Then the dlADMM algorithm is shown in Algorithm 1. Specifically, Lines 5, 6, 10, 11, 14, 15, 17 and 18 solve eight subproblems, namely, , , , , , , and , respectively. Lines 21 and 22 update the residual and the dual variable , respectively.
3.3. The Quadratic Approximation and Backtracking
The eight subproblems in Algorithm 1 are discussed in detail in this section. Most can be solved by quadratic approximation and the backtracking techniques described above, so the operation of the matrix inversion can be avoided.
**1. Update
**The variables are updated as follows:
[TABLE]
The subproblem is transformed into the following form after it is replaced by Equation (1).
[TABLE]
Because and are coupled in , in order to solve this problem, we must compute the inverse matrix of , which involves subiterations and is computationally expensive (taylor2016training, ). In order to handle this challenge, we define as a quadratic approximation of at , which is mathematically reformulated as follows:
[TABLE]
where is a parameter vector, denotes the Hadamard product (the elementwise product), and denotes to the Hadamard power of and is the norm. is the gradient of at . Obviously, . Rather than minimizing the original problem in Equation (2), we instead solve the following problem:
[TABLE]
Because is a quadratic function with respect to , the solution can be obtained by
[TABLE]
given a suitable . Now the main focus is how to choose . Algorithm 2 shows the backtracking algorithm utilized to find a suitable . Lines 2-5 implement a while loop until the condition is satisfied. As becomes larger and larger, is close to and satisfies the loop condition, which precludes the possibility of the infinite loop. The time complexity of Algorithm 2 is , where is the number of features or neurons.
**2. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to the following forms: for ,
[TABLE]
and for ,
[TABLE]
Equation (4) is highly nonconvex because the nonlinear activation function is contained in . For common activation functions such as the Rectified linear unit (ReLU) and leaky ReLU, Equation (4) has a closed-form solution; for other activation functions like sigmoid and hyperbolic tangent (tanh), a look-up table is recommended (taylor2016training, ).
Equation (5) is a convex problem because and are convex with regard to . Therefore, Equation (5) can be solved by Fast Iterative Soft-Thresholding Algorithm (FISTA) (beck2009fast, ).
**3. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to the following form:
[TABLE]
Similarly to the update of , we define as a quadratic approximation of at , which is formulated mathematically as follows (beck2009fast, ):
[TABLE]
where is a parameter. Here for and for are required for the convergence analysis (beck2009fast, ). Without loss of generality, we set , and solve the subsequent subproblem as follows:
[TABLE]
Equation (6) is a convex problem and has a closed-form solution as follows:
[TABLE]
**4. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to the following form:
[TABLE]
Due to the same challenge in updating , we define as a quadratic approximation of at . The quadratic approximation is mathematically reformulated as follows (beck2009fast, ):
[TABLE]
where is a parameter vector, which is chosen by the Algorithm 3. Instead of minimizing the Equation (8), we minimize the following:
[TABLE]
Equation (9) is convex and hence can be solved exactly. If is either an or an regularization term, Equation (9) has a closed-form solution.
**5. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to
[TABLE]
Similarly, we define as a quadratic approximation of at . The quadratic approximation is then mathematically reformulated as follows (beck2009fast, ):
[TABLE]
where is a parameter vector. Instead of minimizing the Equation (10), we minimize the following:
[TABLE]
The choice of is discussed in the supplementary materials111 The supplementary materials are available at http://mason.gmu.edu/~lzhao9/materials/papers/dlADMM_supp.pdf.
**6. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to the following formulation:
[TABLE]
is defined as the quadratic approximation of at as follows:
[TABLE]
where is a parameter. We set for and for , and solve the resulting subproblems as follows:
[TABLE]
. The solutions to Equations (12) and (13) are as follows:
[TABLE]
**7. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to the following forms for :
[TABLE]
and for :
[TABLE]
Solving Equations (14) and (15) proceeds exactly the same as solving Equations (4) and (5), respectively.
**8. Update
**The variables are updated as follows:
[TABLE]
which is equivalent to the following form:
[TABLE]
is defined as the quadratic approximation of at as follows:
[TABLE]
and we can solve the following problem instead:
[TABLE]
where is a parameter vector. The solution to Equation (16) can be obtained by
[TABLE]
To choice of an appropriate is shown in the supplementary materialsfootnote 1.
4. Convergence Analysis
In this section, the theoretical convergence of the proposed dlADMM algorithm is analyzed. Before we formally present the convergence results of the dlADMM algorithms, Section 4.1 presents necessary assumptions to guarantee the global convgerence of dlADMM. In Section 4.2, we prove the global convergence of the dlADMM algorithm.
4.1. Assumptions
Assumption 1 (Closed-form Solution).
There exist activation functions such that Equations (4) and (14) have closed form solutions and , respectively, where and are continuous functions.
This assumption can be satisfied by commonly used activation functions such as ReLU and leaky ReLU. For example, for the ReLU function , Equation (14) has the following solution:
[TABLE]
Assumption 2 (Objective Function).
* is coercive over the nonempty set . In other words, if and . Moreover, is Lipschitz differentiable with Lipschitz constant .*
The Assumption 2 is mild enough for most common loss functions to satisfy. For example, the cross-entropy and square loss are Lipschitz differentiable.
4.2. Key Properties
We present the main convergence result of the proposed dlADMM algorithm in this section. Specifically, as long as Assumptions 1-2 hold, then Properties 1-3 are satisfied, which are important to prove the global convergence of the proposed dlADMM algorithm. The proof details are included in the supplementary materialsfootnote 1.
Property 1 (Boundness).
If , then is bounded, and is lower bounded.
Property 1 concludes that all variables and the value of have lower bounds. It is proven under Assumptions 1 and 2, and its proof can be found in the supplementary materialsfootnote 1.
Property 2 (Sufficient Descent).
*If so that , then there exists
such that*
[TABLE]
Property 2 depicts the monotonic decrease of the objective value during iterations. The proof of Property 2 is detailed in the supplementary materialsfootnote 1.
Property 3 (Subgradient Bound).
There exist a constant and such that
[TABLE]
Property 3 ensures that the subgradient of the objective function is bounded by variables. The proof of Property 3 requires Property 1 and the proof is elaborated in the supplementary materialsfootnote 1. Now the global convergence of the dlADMM algorithm is presented. The following theorem states that Properties 1-3 are guaranteed.
Theorem 4.1.
For any , if Assumptions 1 and 2 are satisfied, then Properties 1-3 hold.
Proof.
This theorem can be concluded by the proofs in the supplementary materialsfootnote 1. ∎
The next theorem presents the global convergence of the dlADMM algorithm.
Theorem 4.2 (Global Convergence).
If , then for the variables in Problem 2, starting from any , it has at least a limit point , and any limit point is a critical point of Problem 2. That is, . Or equivalently,
[TABLE]
Proof.
Because is bounded, there exists a subsequence such that where is a limit point. By Properties 1 and 2, is non-increasing and lower bounded and hence converges. By Property 2, we prove that , , , , , and , as . Therefore , , and , as . Moreover, from Assumption 1, we know that and as . Therefore, . We infer there exists such that as based on Property 3. Specifically, as . According to the definition of general subgradient (Defintion 8.3 in (rockafellar2009variational, )), we have . In other words, the limit point is a critical point of defined in Equation (1). ∎
Theorem 4.2 shows that our dlADMM algorithm converges globally for sufficiently large , which is consistent with previous literature (wang2015global, ; kiaee2016alternating, ). The next theorem shows that the dlADMM converges globally with a sublinear convergence rate .
Theorem 4.3 (Convergence Rate).
*For a sequence
, define , then the convergence rate of is .*
Proof.
The proof of this theorem is included in the supplementary materialsfootnote 1. ∎
5. Experiments
In this section, we evaluate dlADMM algorithm using benchmark datasets. Effectiveness, efficiency and convergence properties of dlADMM are compared with state-of-the-art methods. All experiments were conducted on 64-bit Ubuntu16.04 LTS with Intel(R) Xeon processor and GTX1080Ti GPU.
5.1. Experiment Setup
5.1.1. Dataset
In this experiment, two benchmark datasets were used for performance evaluation: MNIST (lecun1998gradient, ) and Fashion MNIST (xiao2017fashion, ). The MNIST dataset has ten classes of handwritten-digit images, which was firstly introduced by Lecun et al. in 1998 (lecun1998gradient, ). It contains 55,000 training samples and 10,000 test samples with 784 features each, which is provided by the Keras library (chollet2017deep, ). Unlike the MNIST dataset, the Fashion MNIST dataset has ten classes of assortment images on the website of Zalando, which is Europe’s largest online fashion platform (xiao2017fashion, ). The Fashion-MNIST dataset consists of 60,000 training samples and 10,000 test samples with 784 features each.
5.1.2. Experiment Settings
We set up a network architecture which contained two hidden layers with hidden units each. The Rectified linear unit (ReLU) was used for the activation function for both network structures. The loss function was set as the deterministic cross-entropy loss. was set to . was initialized as and was multiplied by 10 every 100 iterations. The number of iteration was set to . In the experiment, one iteration means one epoch.
5.1.3. Comparison Methods
Since this paper focuses on fully-connected deep neural networks, SGD and its variants and ADMM are state-of-the-art methods and hence were served as comparison methods. For SGD-based methods, the full batch dataset is used for training models. All parameters were chosen by the accuracy of the training dataset. The baselines are described as follows:
-
Stochastic Gradient Descent (SGD) (bottou2010large, ). The SGD and its variants are the most popular deep learning optimizers, whose convergence has been studied extensively in the literature. The learning rate of SGD was set to for both the MNIST and Fashion MNIST datasets.
-
Adaptive gradient algorithm (Adagrad) (duchi2011adaptive, ). Adagrad is an improved version of SGD: rather than fixing the learning rate during iteration, it adapts the learning rate to the hyperparameter. The learning rate of Adagrad was set to for both the MNIST and Fashion MNIST datasets.
-
Adaptive learning rate method (Adadelta) (zeiler2012adadelta, ). As an improved version of the Adagrad, the Adadelta is proposed to overcome the sensitivity to hyperparameter selection. The learning rate of Adadelta was set to for both the MNIST and Fashion MNIST datasets.
-
Adaptive momentum estimation (Adam) (kingma2014adam, ). Adam is the most popular optimization method for deep learning models. It estimates the first and second momentum in order to correct the biased gradient and thus makes convergence fast. The learning rate of Adam was set to for both the MNIST and Fashion MNIST datasets.
-
Alternating Direction Method of Multipliers (ADMM) (taylor2016training, ). ADMM is a powerful convex optimization method because it can split an objective function into a series of subproblems, which are coordinated to get global solutions. It is scalable to large-scale datasets and supports parallel computations. The of ADMM was set to for both the MNIST and Fashion MNIST datasets.
5.2. Experimental Results
In this section, experimental results of the dlADMM algorithm are analyzed against comparison methods.
5.2.1. Convergence
First, we show that our proposed dlADMM algorithm converges when is sufficiently large and diverges when is small for both the MNIST dataset and the Fashion MNIST dataset.
The convergence and divergence of dlADMM algorithm are shown in Figures 2 and 3 when and ,respectively. In Figures 2(a) and 3(a), the X axis and Y axis denote the number of iterations and the logarithm of objective value, respectively. In Figures, 2(b) and 3(b), the X axis and Y axis denote the number of iterations and the logarithm of the residual, respectively. Figure 2, both the objective value and the residual decreased monotonically for the MNIST dataset and the Fashion-MNIST dataset, which validates our theoretical guarantees in Theorem 4.2. Moreover, Figure 3 illustrates that both the objective value and the residual diverge when . The curves fluctuated drastically on the objective value. Even though there was a decreasing trend for the residual, it still fluctuated irregularly and failed to converge.
5.2.2. Performance
Figure 4 and Figure 5 show the curves of the training accuracy and test accuracy of our proposed dlADMM algorithm and baselines, respectively. Overall, both the training accuracy and the test accuracy of our proposed dlADMM algorithm outperformed most baselines for both the MNIST dataset and the Fashion MNIST dataset. Specifically, the curves of our dlADMM algorithm soared to at the early stage, and then raised steadily towards to or more. The curves of the most SGD-related methods, SGD, Adadelta, and Adagrad, moved more slowly than our proposed dlADMM algorithm. The curves of the ADMM also rocketed to around , but decreased slightly later on. Only the state-of-the-art Adam performed better than dlADMM slightly.
5.2.3. Scalability Analysis
In this subsection, the relationship between running time per iteration of our proposed dlADMM algorithm and three potential factors, namely, the value of , the size of training samples, and the number of neurons was explored. The running time was calculated by the average of 200 iterations.
Firstly, when the training size was fixed, the computational result for the MNIST dataset and Fashion MNIST dataset is shown in Table 2. The number of neurons for each layer ranged from 200 to 1,000, with an increase of 200 each time. The value of ranged from to , with multiplying by 10 each time. Generally, the running time increased as the number of neurons and the value of became larger. However, there were a few exceptions: for example, when there were 200 neurons for the MNIST dataset, and increased from to , the running time per iteration dropped from seconds to seconds.
Secondly, we fixed the number of neurons for each layer as . The relationship between running time per iteration, the training size and the value of is shown in Table 3. The value of ranged from to , with multiplying by 10 each time. The training size of the MNIST dataset ranged from to , with an increase of each time. The training size of the Fashion MNIST dataset ranged from to , with an increase of each time. Similiar to Table 3, the running time increased generally as the training sample and the value of became larger and some exceptions exist.
6. Conclusion and Future Work
Alternating Direction Method of Multipliers (ADMM) is a good alternative to Stochastic Gradient Descent (SGD) for deep learning problems. In this paper, we propose a novel deep learning Alternating Direction Method of Multipliers (dlADMM) to address some previously mentioned challenges. Firstly, the dlADMM updates parameters from backward to forward in order to transmit parameter information more efficiently. The time complexity is successfully reduced from to by iterative quadratic approximations and backtracking. Finally, the dlADMM is guaranteed to converge to a critical solution under mild conditions. Experiments on benchmark datasets demonstrate that our proposed dlADMM algorithm outperformed most of the comparison methods.
In the future, we may extend our dlADMM from the fully-connected neural network to the famous Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN), because our convergence guarantee is also applied to them. We also consider other nonlinear activation functions such as sigmoid and hyperbolic tangent function (tanh).
Acknowledgement
This work was supported by the National Science Foundation grant: .
Appendix A Algorithms to update and
The algorithms to update and are described in the Algorithms 4 and 5, respectively.
Appendix B Lemmas for the Proofs of Properties
The following several lemmas are preliminary results.
Lemma B.1.
Equation (9) holds if and only if there exists , the subgradient of such that
[TABLE]
Likewise, Equation (11) holds if and only if there exists , the subgradient of such that
[TABLE]
Proof.
These can be obtained by directly applying the optimality conditions of Equation (9) and Equation (11), respectively. ∎
Lemma B.2.
* for all .*
Proof.
The optimality condition of in Equation (15) gives rise to
[TABLE]
Because , then we have . ∎
Lemma B.3.
It holds that ,
[TABLE]
Proof.
Because is Lipschitz differentiable by Assumption 2, so is . Therefore, this lemma is proven exactly as same as Lemma 2.1 in [1]. ∎
Lemma B.4.
For Equations (6) and (12), if ,then the following inequalities hold:
[TABLE]
Proof.
Because is Lipschitz differentiable with respect to b with Lipschitz coefficient (the definition of Lipschitz differentiablity can be found in [1]), we directly apply Lemma 2.1 in [1] to to obtain Equations (19) and (20), respectively. ∎
Lemma B.5.
It holds that for ,
[TABLE]
Proof.
Essentially, all inequalities can be obtained by applying optimality conditions of updating , , , , , , and , respectively. We only prove Inequality (23), (27), (28) and (30) .This is because Inequalities (21), (26) and (31) follow the routine of Inequality (27), Inequalities (24), (25) and (29) follow the routine of Inequality (28), and Inequality (22) follows the routine of Inequality (30).
Firstly, we focus on Inequality (23).
[TABLE]
where the second equality follows from the cosine rule .
According to the optimality condition of Equation (5), we have . Because is convex and differentiable with regard to , its subgradient is also its gradient. According to the definition of subgradient, we have
[TABLE]
We introduce Equation (33) into Equation (32) to obtain Equation (23).
Secondly, we focus on Inequality (27). The stopping criterion of Algorithm 4 shows that
[TABLE]
Because is convex, according to the definition of subgradient, we have
[TABLE]
where is defined in the premise of Lemma B.1. Therefore, we have
[TABLE]
Thirdly, we focus on Inequality (28).
[TABLE]
Finally, we focus on Equation (30). This follows directly from the optimality of in Equation (14). ∎
Lemma B.6.
If so that , then it holds that
[TABLE]
Proof.
[TABLE]
We choose to make . ∎
Appendix C Proof of Theorem 4.1
Proving Theorem 4.1 is equal to proving jointly Theorem C.1, C.2, and C.3, which are elaborated in the following.
Theorem C.1.
Given that Assumptions 1 and 2 hold, the dlADMM satisfies Property 1.
Proof.
There exists such that . By Assumption 2, we have
[TABLE]
where . Then we have
[TABLE]
It concludes from Lemma B.5 and Lemma B.6 that is upper bounded by and so are and . By Assumption 2, is bounded. By Lemma B.2, it is obvious that is bounded as well. ∎
Theorem C.2.
Given that Assumptions 1 and 2 hold, the dlADMM satisfies Property 2.
Proof.
This follows directly from Lemma B.5 and Lemma B.6. ∎
Theorem C.3.
Given that Assumptions 1 and 2 hold, the dlADMM satisfies Property 3.
Proof.
We know that
[TABLE]
where , , , and . To prove Property 3, we need to give an upper bound of and by a linear combination of , , , and .
For ,
[TABLE]
Because
[TABLE]
and the optimality condition of Equation (11) yields
[TABLE]
Because is bounded by Property 1, can be upper bounded by a linear combination of , and .
For ,
[TABLE]
Because
[TABLE]
and the optimality condition of Equation (11) yields
[TABLE]
Because is bounded by Property 1, can be upper bounded by a linear combination of , , and .
For ,
[TABLE]
The optimality condition of Equation (12) yields
[TABLE]
Therefore, is linearly independent on .
For ,
[TABLE]
Because
[TABLE]
and the optimality condition of Equation (13) yields
[TABLE]
Therefore, is upper bounded by a combination of and .
For ,
[TABLE]
because is bounded and is continuous and hence is bounded, and the optimality condition of Equation (14) yields
[TABLE]
Therefore, is upper bounded by .
For ,
[TABLE]
The optimality condition of Equation (15) yields
[TABLE]
and
[TABLE]
Therefore is upper bounded by .
For ,
[TABLE]
Because
[TABLE]
Then we need to show that , , and are upper bounded by , , , and .
[TABLE]
Because and are upper bounded, is therefore upper bounded by a combination of and .
Similarly, is upper bounded by a combination of and .
[TABLE]
Because , and are upper bounded, is therefore upper bounded by a combination of and .
For ,
[TABLE]
Because
[TABLE]
Then we need to show that , , and are upper bounded by , , , and .
[TABLE]
Because and are upper bounded, is therefore upper bounded by a combination of and .
Similarly, is upper bounded by a combination of and .
[TABLE]
Because and are bounded, is upper bounded by a combination of and .
[TABLE]
Because , and are upper bounded, is therefore upper bounded by a combination of and .
For ,
[TABLE]
Because
[TABLE]
Therefore, is upper bounded by . ∎
Appendix D Proof of Theorem 4.3
Proof.
To prove this theorem, we will first show that satisfies two conditions: (1). . (2). is bounded. We then conclude the convergence rate of based on these two conditions. Specifically, first, we have
[TABLE]
Therefore satisfies the first condition. Second,
[TABLE]
So is bounded and satisfies the second condition. Finally, it has been proved that the sufficient conditions of convergence rate are: (1) , and (2) is bounded, and (3) (Lemma 1.2 in [6]). Since we have proved the first two conditions and the third one is obvious, the convergence rate of is proven. ∎
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1(1) Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences , 2(1):183–202, 2009.
- 2(2) Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010 , pages 177–186. Springer, 2010.
- 3(3) Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning , 3(1):1–122, 2011.
- 4(4) Caihua Chen, Min Li, Xin Liu, and Yinyu Ye. Extended admm and bcd for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights. Mathematical Programming , pages 1–41, 2015.
- 5(5) Francois Chollet. Deep learning with python . Manning Publications Co., 2017.
- 6(6) Wei Deng, Ming-Jun Lai, Zhimin Peng, and Wotao Yin. Parallel multi-block admm with o (1/k) convergence. Journal of Scientific Computing , 71(2):712–736, 2017.
- 7(7) John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(Jul):2121–2159, 2011.
- 8(8) Yuyang Gao, Liang Zhao, Lingfei Wu, Yanfang Ye, Hui Xiong, and Chaowei Yang. Incomplete label multi-task deep learning for spatio-temporal event subtype forecasting. 2019.
