Parabolic Approximation Line Search for DNNs
Maximus Mutschler, Andreas Zell
TL;DR
This paper introduces a parabolic approximation line search method for deep neural network optimization that adaptively finds effective step sizes by exploiting the local convexity of loss functions, improving over traditional methods.
Contribution
It presents a novel line search technique based on parabolic approximation that automatically adjusts step sizes in DNN training, eliminating the need for hand-designed schedules.
Findings
Outperforms other step size estimation methods in experiments.
Competitively matches common optimization algorithms across various datasets.
Shows that local loss landscapes are often convex and suitable for parabolic approximation.
Abstract
A major challenge in current optimization research for deep learning is to automatically find optimal step sizes for each update step. The optimal step size is closely related to the shape of the loss in the update step direction. However, this shape has not yet been examined in detail. This work shows empirically that the batch loss over lines in negative gradient direction is mostly convex locally and well suited for one-dimensional parabolic approximations. By exploiting this parabolic property we introduce a simple and robust line search approach, which performs loss-shape dependent update steps. Our approach combines well-known methods such as parabolic approximation, line search and conjugate gradient, to perform efficiently. It surpasses other step size estimating methods and competes with common optimization methods on a large variety of experiments without the need of…
| network | seconds / epoch PAL | SLS | SGD | ALIG | SGDHD | COCOB |
|---|---|---|---|---|---|---|
| ResNet32 | ||||||
| MobilenetV2 | ||||||
| EfficientNet | ||||||
| DenseNet40 |
| dataset | network | optimizer | training loss | validation accuracy | test accuracy | |||
| min | median; p25; p75 | max | median; p25; p75 | max | median; p25; p75 | |||
| CIFAR-10 | EfficientNet | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-10 | MobileNetV2 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-10 | DenseNet40 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-10 | ResNet32 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-100 | DenseNet40 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-100 | EfficientNet | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-100 | MobileNetV2 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| CIFAR-100 | ResNet32 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| TOLSTOI | RNN | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| SLS | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| ImageNet | ResNet50 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SLS | ||||||||
| SGD | ||||||||
| PAL | ||||||||
| ImageNet | DenseNet121 | COCOB | ||||||
| ALIG | ||||||||
| SGDHD | ||||||||
| RMSP | ||||||||
| ADAM | ||||||||
| SLS | ||||||||
| SGD | ||||||||
| PAL | ||||||||
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsMachine Learning and Data Classification · Machine Learning and Algorithms · Domain Adaptation and Few-Shot Learning
Parabolic Approximation Line Search for DNNs
Maximus Mutschler and Andreas Zell
University of Tübingen
Sand 1, D-72076 Tübingen, Germany
{maximus.mutschler, andreas.zell}@uni-tuebingen.de
Abstract
A major challenge in current optimization research for deep learning is automatically finding optimal step sizes for each update step. The optimal step size is closely related to the shape of the loss in the update step direction. However, this shape has not yet been examined in detail. This work shows empirically that the mini-batch loss along lines in negative gradient direction is locally mostly convex and well suited for one-dimensional parabolic approximations. We introduce a simple and robust line search approach by exploiting this parabolic observation, which performs loss-shape-dependent update steps. Our approach combines well-known methods such as parabolic approximation, line search, and conjugate gradient to perform efficiently. It surpasses other step size estimating methods and competes with standard optimization methods on a large variety of experiments without the need for hand-designed step size schedules. Thus, it is of interest for objectives where step-size schedules are unknown or do not perform well. Our extensive evaluation includes multiple comprehensive hyperparameter grid searches on several datasets and architectures. Finally, we provide a general investigation of exact line searches in the context of batch losses and exact losses, including their relation to our line search approach.
1 Introduction
Automatic determination of optimal step sizes for each update step of stochastic gradient descent is a major challenge in current optimization research for deep learning [52, 5, 38, 29, 45, 60, 12, 3, 48]. One default approach to tackle this challenge is to apply line search methods. Several of these have been introduced for Deep Learning [38, 29, 45, 60, 12]. However, these approaches have not analyzed the shape of the loss functions in update step direction in detail, which is important since the optimal step size stands in strong relation to this shape. To shed light on this, our work empirically analyses the shape of the loss function along update step direction for deep learning scenarios often considered in optimization. We further elaborate on the properties found to define a simple and empirically justified optimizer.
Our contributions are as follows: 1: Our empirical analysis suggests that the mini-batch loss in the negative gradient direction mostly shows locally convex shapes. Furthermore, we show that parabolic approximations are well suited to estimate the minima in these directions (Section 3). 2: Exploiting this parabolic observation, we build a simple line search optimizer that constructs its loss function-dependent learning rate schedule. The performance of our optimization method is extensively analyzed, including a comprehensive comparison to other optimization methods (Sections 4,5). 3: We provide a convergence analysis that backs our empirical results under strong assumptions (Section 4.4). 4: We provide a general investigation of exact line searches on batch losses and their relation to line searches on the exact loss as well as their relation to our line search approach (Section 6) and, finally, analyze the relation of our approach to interpolation (Section 7).
The full-batch loss is defined as the average over realizations of a mini-batch-wise loss function : , with being a batch of input samples, and the amount of batches in the dataset. denotes the parameters to be optimized. Note, that we consider a sample as one mini-batch of multiple inputs. In this work, we consider at optimization step in negative gradient direction:
[TABLE]
where is . We denote as mini-batch loss along a line and is the step on this line. The motivation of our work builds upon the following assumption:
Assumption 1**.**
(Informal) The position of a minimum of is a well enough estimator for the position of a minimum of the full-batch loss on the same line to perform a successful optimization process.
We empirically analyze Assumption 1 further in section 6.
2 Related work
Our optimization approach is based on well-known methods, such as line search, the non-linear conjugate gradient method, and quadratic approximation, which can be found in Numerical Optimization [28]. In addition, [28, §3.5] describes a similar line search routine for the deterministic setting. The concept of parabolic approximations is also exploited by the well-known line search of More and Thunte [40]. Our work contrasts common optimization approaches in deep learning by directly exploiting the local parabolic observation (see Section 3) of . Similarly, SGD-HD [3] performs update steps towards a minimum of , by performing gradient descent on the learning rate. Concurrently, [10] explored a similar direction as this work by analyzing possible line search approximations for DNN loss landscapes but does not exploit these for optimization.
The recently published Stochastic Line-Search (SLS) [60] is an optimized backtracking line search based on the Armijo condition, which samples, like our approach, additional mini-batch losses from the same batch and checks the Armijo condition on these. [60] assumes that the model interpolates the data. The latter states formally that the gradient with respect to each batch converges to zero at the optimum of the convex function. This implies that if is minimized at and thus holds, then for all we have that . SLS exhibits competitive performance against multiple optimizers on several DNN tasks. [45] introduces a related idea but does not provide empirical results for DNNs. [12] also uses a backtracking Armijo line search, but with the aim to regulate the optimal batch size.
The methodically appealing but complex Probabilistic Line Search (PLS) [38] approximates along lines with a Gaussian Process posterior based on realizations of for different batches . The mean of this Posterior is a cubic spline. A suitable update step size is found by exploiting a probabilistic formulation of the Wolf conditions.
Gradient Only Line Search (GOLS1) [29] GOLS1 searches for a minimum on lines by looking for a sign change of the first directional mini-batch derivative in search direction. For each measurement, a different is chosen. Both approaches show that they can optimize successfully on several machine learning problems and can compete against plain SGD.
From the perspective of assumptions about the shape of the loss landscape, second order methods such as oLBFGS [55], KFRA [7], L-SR1 [47], QUICKPROP [15], S-LSR1 [4], and KFAC [39] generally assume that the loss function can be approximated locally by a parabola of the same dimension as the loss function. Adaptive methods such as SGD with momentum [51], ADAM[30], ADAGRAD[14], ADABOUND[37], AMSGRAD[49] or RMSProp [59] focus more on the handling of noise than on shape assumptions. In addition, methods exist that approximate the loss function in specific directions: The L4 adaptation scheme [52] as well as ALIG [5] estimate step sizes by approximating the loss function linearly in negative gradient direction, whereas our approach approximates the loss function parabolically in negative gradient direction. Finally, COCOB [44] has to be mentioned, an alternative learning rate free approach, which automatically estimates step directions and sizes with a reward-based coin betting concept.
3 Empirical analysis of the shape of mini-batch losses along lines
In this section, we analyze mini-batch losses along lines (see Eq. 1) during the training of multiple architectures and show that they locally exhibit mostly convex shapes, which are well suited for parabolic approximations. We focus on CIFAR-10, as it is extensively analyzed in optimization research for deep learning. However, we observed similar results on MNIST, CIFAR-100, and ImageNet, considering random samples. We analyzed mini-batch losses along the lines in the first 10000 SGD update step directions for four commonly used architectures (ResNet-32 [23], DenseNet-40 [25], EfficientNet [58], MobileNetV2 [54]). Along each line, we sampled 50 losses and performed a parabolic approximation (see Section 4). An unbiased selection of our results on a ResNet32 is shown in Figure 1. Further results are given in Appendix A. In accordance with [61], we conclude that the mini-batch losses along the analyzed lines tend to be locally convex. In addition, one-dimensional parabolic approximations of the form with are well suited to estimate the position of a minimum along such directions.
To substantiate the later observation, we analyzed the angle between the line direction and the gradient at the -by a parabola- estimated minimum during training. A position is a local extremum or saddle point of the loss along a line if and only if the angle between the line direction and the gradient at the position is , if measured on the same mini-batch. 111This holds because if the directional derivative of the measured gradient in line direction is 0, the current position is an extremum or saddle point along the line and the angle is . If the position is not an extremum or saddle point, the directional derivative is not 0 [28]. Measuring step sizes and update step adaptations factors (see Sections 4.1 and4.3) were chosen to fit the mini-batch loss along the line decently. As shown in Figures 2 and 3, the parabolic observation holds well for several architectures trained on MNIST, CIFAR-10, CIFAR-100 and ImageNet. The observation fits best for MNIST and gets worse for more complex tasks such as ImageNet. We can ensure that the extrema found are minima since we additionally plotted the mini-batch loss along the line for each update step.
In addition, we analyzed the mini-batch loss along lines in conjugate-like directions and random directions. Mini-batch losses along lines in conjugate like directions also tend to have convex shapes (see Appendix D.4 Figure 17 ). However, mini-batch losses along lines in random directions rarely exhibit convex shapes.
4 The line search algorithm
We introduce Parabolic Approximation Line Search (PAL), which exploits the observation that parabolic approximations are suited to estimate the minimizer of the mini-batch loss along lines. This simple approach combines well-known methods from basic optimization such as parabolic approximation and line search [28] to perform an efficient line search. We note that the general idea of this method can be applied to any optimizer that provides an update step direction.
4.1 Parameter update rule
An intuitive explanation of PAL’s parameter update rule based on a parabolic approximation is given in Figure 4. Since (see Eq.1) is assumed to exhibit a convex and almost parabolic shape, we approximate it with with and . Consequently, we need three measurements of to define and . Those are given by the current loss , the derivative
in gradient direction (see Eq. 4) and an additional loss with measuring distance . It is simple to show that , , and . The update step to the minimum of the parabolic approximation is thus given by:
[TABLE]
Note, that is the second derivative of the approximated parabola and is only identical to the exact directional derivative if the parabolic approximation fits. The normalization of the gradient to unit length (Eq.1) was chosen to have the measuring distance independent of the gradient size and of weight scaling. Note that two network inferences are required to determine and . Consequently, PAL needs two forward passes and one backward pass through a model. Further on, may include random components, but to ensure continuity during one line search, drawn random numbers have to be reused for each value determination of at (e.g., for Dropout [57]). The memory required by PAL is similar to SGD with momentum, since only the last update direction has to be saved. A basic version of PAL is given in Algorithm 1.
4.2 Case discrimination of parabolic approximations
Since not all parabolic approximations are suitable for parameter update steps, the following cases are considered separately. Note that and .. 1: and : the parabolic approximation has a minimum in line direction, thus, the parameter update is done as described in Section 4.1. 2: and : the parabolic approximation has a maximum in negative line direction or is a line with a negative slope. In those cases, a parabolic approximation is inappropriate. is set to , since the second measured point has a lower loss than the first. 3: Since cannot be greater than 0, the only case left is an extremum at the current position (. In this case, no weight update is performed. However, the mini-batch loss function changes with the next batch, which likely does not have an extremum at exactly the same point. In accordance with Section 3, cases two and three appeared very rarely in our experiments.
4.3 Additions
Here multiple additions for Algorithm 1 to fine-tune the performance and handle degenerate cases are introduced. Our hyperparameter sensitivity analysis (Appendix D.6) suggests that the influence of the introduced hyperparameters on the optimizer’s performance is low. Thus, they only need to be adapted to fine-tune the results. The full version of PAL including all additions is given in Appendix B Algorithm 2.
Direction adaptation: Instead of following the direction of the negative gradient we follow an adapted conjugate-like direction :
[TABLE]
with . Since now an adapted direction is used, changes to:
[TABLE]
This approach aims to find a more optimal search direction than the negative gradient. We implemented and tested the formulas of Fletcher-Reeves [16], Polak-Ribière [50], Hestenes-Stiefel [24] and Dai-Yuan [11] to determine conjugate directions under the assumption that the loss function is a quadratic. However, choosing a constant of value or performs equally. The influence of and dynamic update steps on PAL’s performance is discussed in Appendix D.5. In the analyzed scenario, can both increase and decrease the performance, whereas dynamic update steps mostly increase the performance. A combination of both is needed to achieve optimal results.
Update step adaptation: Our preliminary experiments revealed a systematic error caused by constantly approximating with slightly too narrow parabolas. Therefore, is multiplied by a parameter (compare to Eq. 2). This is useful to estimate the minimum’s position along a line more exactly but has minor effects on training performance.
Maximum step size: To hinder the algorithm from failing due to inaccurate parabolic approximations, we use a maximum step size . The new update step is given by . However, most of our experiments with never reached this step size and still performed well.
4.4 Theoretical considerations
Usually, convergence in deep learning is shown for convex stochastic functions with an L-Lipschitz continuous gradient. However, since our approach originates from empirical results, it is not given that a profound theoretical analysis is possible. In order to show any convergence guarantees for parabolic approximations, we have to fall back to uncommonly strong assumptions, which lead to quadratic models. Since convergence proofs on quadratics are of minor importance for most readers, our derivations can be found in Appendix C.
5 Evaluation
5.1 Experimental design
We performed a comprehensive evaluation to analyze the performance of PAL on a variety of deep learning optimization tasks. Therefore, we tested PAL on commonly used architectures on CIFAR-10 [31], CIFAR-100 [31] and ImageNet [13]. For CIFAR-10 and CIFAR-100, we evaluated on DenseNet40 [25], EfficientNetB0 [58], ResNet32 [23] and MobileNetV2 [54]. On ImageNet we evaluated on DenseNet121 and ResNet50. In addition, we considered an RNN trained on the Tolstoi war and peace text prediction task. We compare PAL to SLS[60], whose Armijo variant is state-of-the-art in the line search field for DNNs at the time of writing. In addition, we compare against the following well studied and widely used first order optimizers: SGD with momentum [51], ADAM[30], and RMSProp[59] as well as against SGDHD [3] and ALIG [5], which automatically estimate learning rates in negative gradient direction and, finally, against the coin betting approach COCOB [44]. To perform a fair comparison, we compared various hyperparameter combinations of commonly used hyperparameters for each optimizer. In addition, we utilize those combinations to analyze the hyperparameter sensitivity for each optimizer. Since a grid search on Imagenet was too expensive, the best hyperparameter configuration of CIFAR-100’s evaluation was used to test hyperparameter transferability. A detailed explanation of the experiments including hyperparameters and data augmentations used, are given in Appendix D.8. All in all, we trained over 4500 networks with Tensorflow 1.15 [1] on Nvidia Geforce GTX 1080 TI graphic cards. Since PAL is a line search approach, the predefined learning rate schedules of SGD and the generated schedules of SLS, ALIG, SGDHD and PAL were compared. Due to normalization, PAL’s learning rate is given by .
5.2 Results
A selection of our results is given in Figure 5. The results of other architectures trained on CIFAR-10, CIFAR-100, Imagenet and Tolstoi are found in Appendix D Figures 13,14,15. A table with exact numerical results of all experiments is provided in Appendix D.9.
In most cases PAL decreases the training loss faster and to a lower value than the other optimizers (row 1 of Figures 5,13,14,15). Considering validation and test accuracy, PAL surpasses ALIG, SGDHD and COCOB, competes with RMSProp and ADAM but gets surpassed by SGD (rows 2,3 of Figures 5,13,14,15). However, RMSProp, ADAM and SGD were tuned with a step size schedule. If we compare PAL to their basic implementations without a schedule, which roughly corresponds to the first plateau reached in row 2 of Figures 5,13,14,15, PAL would surpass the other optimizers and shows that it can find a well performing step size schedule. This is especially interesting for problems for which default schedules might not work.
SLS decreases the training loss further than the other optimizers on a few problems but shows weak performance and poor generalization on most. This contrasts to the results of [60], where SLS behaves robustly and excels. To exclude the possibility of errors on our side, we reimplemented SLS experiment on ResNet34 and could reproduce a similar well performance as in [60] (Appendix D.3). Our results suggest that the interpolation assumption on which SLS is based is not always valid for the considered tasks.
Considering the box plots of Figures 5 and 14, which represent the sensitivity to hyperparameter combinations, one would likely try on a new unknown objective, we can see that PAL has a strong tendency to exhibit low sensitivity in combination with good performance. To emphasize this statement, a sensitivity analysis of PAL’s hyperparameters (Appendix Figure 19) shows that PAL performs well on a wide range for each hyperparameter on a ResNet32.
On wall-clock-time PAL performs as fast as SLS but slower than the other optimizers, which achieve similar speeds (Appendix D.2). However, an automatic, well-performing learning rate schedule might compensate for the slower speed depending on the scenario.
Considering the learning rate schedules of PAL (row 4 of Figures 5,13,14,15) we achieved unexpected results. PAL, which estimates the learning rate directly from approximated local shape information, does not follow a schedule that is similar to the one of SLS, ALIG, SGDHD or any of the commonly used handcrafted schedules such as piece-wise constant or cosine decay. However, it achieves similar results. An interesting side result is that ALIG and SGDHD tend to perform best if hyperparameters are chosen in a way that the learning rate is only changed slightly and, therefore, virtually an SGD training with a fixed learning rate is performed.
6 On the exactness of line searches on mini-batch losses
In this section, we investigate the general question of whether line searches that estimate the position of a minimum of mini-batch losses exactly are beneficial. In Figure 2 we showed that PAL can perform an almost exact line search on batch losses if we use a fixed update step adaptation factor (Section 4.3). However, PAL’s best hyperparameter configuration does not perform an exact line search (see Figure 6). Consequently, we analyzed how an exact line search, which exactly estimates a minimum along the line, behaves. We implemented an inefficient binary line search (see Appendix E), which measured up to 20 values on each line to estimate the position of a minimum. The results, given in Figure 6, show that an optimal line search does not optimize well. Thus, the reason why PAL performs well is not the exactness of its update steps. In fact, slightly inexact update steps seem to be beneficial.
These results query Assumption 1, which assumes that the position of a minimum on a line in negative gradient direction of is a suitable estimator for the minimum of the full-batch loss on this line to perform a successful optimization process. To investigate this further, we measured and the distribution of mini-batch losses for multiple SGD update directions on a ResNet32. Our results suggest, as exemplarily shown in Figure 7 that the position of ’s minimum along a line in negative gradient direction is not always a good estimator for the position of ’s corresponding minimum. This explains why exact line searches on the batch loss perform weakly.
Corollaries are that the empirical loss on the investigated lines also tends to be locally convex and that the optimal step size tends to be smaller than the step size given by the batch loss on such lines. This is a possible explanation why the slightly too narrow parabolic approximations of PAL without update step adaptation perform well.
7 PAL and Interpolation
This section analyzes whether the reason why PAL performs well is related to the interpolation condition. Formally, interpolation requires that the gradient with respect to each sample converges to zero at the optimum. We repeated the experiments of the SLS paper (see [60] Section 7.2 and 7.3), which analyze the performance on problems for which interpolation hold or does not hold.
Figure 8 shows that PAL such as SLS converge faster to an artificial optimization floor on non-over-parameterized models () of the matrix factorization problem of [60, §7.2]. In the interpolation case PAL and SLS converge linearly to machine precision. On the binary classification problem of [60] Section 7.3, which uses a softmax loss and RBF kernels on the mushrooms and ijcnn datasets, we observe that PAL and SLS converge fast on the mushrooms task, for which the interpolation condition holds (Figure 9). However, PAL converges faster on the ijcnn task, for which the interpolation condition does not hold.
The results indicate that the interpolation condition is beneficial for PAL, but, PAL performs also robust when it is likely not satisfied (see Figure 5,13,14,15. In those experiments PAL mostly performs competitive but SLS does not. However, the relation of the parabolic observation to interpolation needs to be investigated more closely in future.
8 Conclusions
This work tackles a major challenge in current optimization research for deep learning: to automatically find optimal step sizes for each update step. In detail, we focus on line search approaches to deal with this challenge. We introduced a robust and straightforward line search approach based on one-dimensional parabolic approximations of mini-batch losses. The introduced algorithm is an alternative to SGD for objectives where default decays are unknown or do not work.
Loss functions of DNNs are commonly perceived as being highly non-convex. Our analysis suggests that this intuition does not hold locally since lines of loss landscapes across models and datasets can be approximated parabolically to high accuracy. This new knowledge might further help to explain why update steps of specific optimizers perform well.
To gain deeper insights into line searches in general, we analyzed how an expensive but exact line search on batch losses behaves. Intriguingly, its performance is weak, which lets us conclude that the minor inaccuracies of the parabolic approximations are beneficial for training.
Retrospective from 10.2021
After the publication of this paper, we conducted further research in this field and observed the following: PAL’s performance is highly dependent on the weight initialization and batch size. In general, it does not work well with Pytorch’s default weight initialization. Note that the experiments here were done with Tensorflow 1.5. In addition, it is hard to find hyperparameters that compete with SGD. Our further works exploring the empirics-based line search direction are [42, 43].[42] performs a comprehensive empirical study of the shape of the full-batch loss in line direction and the performance of several optimizers along such lines. [43] introduces a line search approach that fits the full-batch loss with a parabola and can also handle smaller batch sizes.
Ethics Statement
Since we understand our work as basic research, it is highly error-prone to estimate its specific ethical aspects and future positive or negative social consequences. As optimization research influences the whole field of deep learning, we refer to the following works, which discuss the ethical aspects and social consequences of AI and Deep Learning in a comprehensive and general way:[63, 41, 6].
Acknowledgments
Maximus Mutschler heartly thanks Lydia Federmann, Kevin Laube, Jonas Tebbe, Mario Laux, Valentin Bolz, Hauke Neitzel, Leon Varga, Benjamin Kiefer, Timon Höfer, Martin Meßmer, Cornelia Schulz, Hamd Riaz, Nuri Benbarka, Samuel Scherer, Frank Schneider, Robert Geirhos and Frank Hirschmann for their comprehensive support.
Funding
This research was supported by the German Federal Ministry of Education and Research (BMBF) project ’Training Center Machine Learning, Tübingen’ with grant number 01|S17054.
Appendix A Further line plots
Appendix B PAL with all additions:
Algorithm 2 provides the full description of PAL including the additions described in Section 4.3. After analyzing the best hyperparameter combinations of PAL over all experiments, we suggest to use values from the following parameter intervals: , , , . Where and usually have a low sensitivity. Thus, the basic implementation of PAL (Algorithm 1) performs already well and is always found in the upper quartile considering the performance of all analyzed hyperparameter configurations in our experiments. PyTorch and Tensorflow 1.5 implementations are provided at https://github.com/cogsys-tuebingen/PAL.
Appendix C Further Theoretical Considerations
We have to note that the following derivation is based upon strong assumptions and if they are valid at all, they are likely more valid locally than globally. We assume that each slice of the loss function is a one-dimensional parabolic function:
Assumption 2**.**
Let be the number of parameters and let be vectors. Then for all there exists with , such that for all
This strong assumption is a simplified adaptation to our empirical results that lines in negative gradient direction behave locally almost parabolic (see Section 3). For the following derivations we assume a basic PAL without the additions introduced in Section 4.3. Proofs are provided in Appendix C.1. At first we show that is a n-dimensional parabolic function:
Lemma 1**.**
Let be a k-times continuously differentiable function. Furthermore, assume there exists with , such that for all . Then there exist and a positive definite Matrix such that for all .
Now we show that PAL converges on :
Proposition 1**.**
PAL converges on with hermitian and positive definite.
We have to note that in this scenario, PAL is identical to the method of steepest descent for which the convergence, including convergence rates on quadratics, is already proven in [36, Page 235]. Nevertheless, we have attached our proof better adapted to this scenario.
For a noisy scenario where each batch defines a quadratic, PAL has no convergence guarantee. Given two shifted one-dimensional parabolas, and , which are presented to PAL alternately, PAL will always perform an update step to the minimum position of one of these but never to the minimum position of the average of both. By slightly changing the training procedure and assuming that each has the same this can be fixed:
Proposition 2**.**
If and with being number the of batches . (Each batch defines a parabola. The empirical loss is the mean of these parabolas). And for all it holds that and that is positive definite. Then holds.
This implies that under Assumption 2 and a fixed the position of the minimum of the empirical loss is given by the average of the positions of the mini-batch losses’ minima. The minimum position of full-batch loss is found by PAL, by slightly adapting PAL to search on one batch until it finds the minimum’s position and then averaging the minimum of each batch. As a result, PAL converges in this noisy scenario. However, we have to emphasize at this point that our assumptions about and are likely not valid for general deep learning scenarios. Nevertheless, if it is locally valid, this direction might be a further explanation, in addition to those of [22, 17], why averaging the weights of each update step in the last epoch (stochastic weight averaging) [27] performs well.
C.1 Proofs
Lemma 1**.**
Let be a k-times continuously differentiable function. Furthermore, assume there exists with , such that for all . Then there exist and a positive definite Matrix such that for all .
Proof.
[TABLE]
holds since we have a polynomial of degree 2 and its third derivative is always a tensor.
holds since the reminder of the quadratic Taylor expansion is always 0.
In our case the right part is 0 since:
[TABLE]
In words: is a parabolic function if and only if for each location the third directional derivative of in each direction is [math]. Which is the case, since the third derivative of each intersection is 0.
is positive definite since:
[TABLE]
where is the Hessian. ∎
Proposition 1**.**
PAL converges on with hermitian and positive definite.
Proof.
For this prove we consider a basic PAL without the features introduced in Section 4.3. Note, that during the proof we will see, that and . Thus, only the update step for this case has to be considered (see Section 4.2.
is convex since is positive definite. Thus it has one minimum.
Without loss of generality we set
[TABLE]
The values of along a line through in the direction of are given by:
[TABLE]
Now we expand the line function:
[TABLE]
Here we see that is indeed a parabolic function with , and since , and are positive definite.
The location of the minimum of is given by:
[TABLE]
PAL determines exactly with (see equation 1 and 2). since otherwise we are already in the minimum.
The value at the minimum is given by:
[TABLE]
Since and are positive definite and :
[TABLE]
Now we consider the sequence , with defined by PAL (see Equation 1):
[TABLE]
It is easily seen by induction that:
[TABLE]
converges to 0. Since and is bounded.
Now we have to show that converges to 0.
We have:
[TABLE]
Now we use the theorem of Courant-Fischer:
[TABLE]
And get:
[TABLE]
with
[TABLE]
Thus, we have:
[TABLE]
Since converges to 0, converges to 0.
This means, converges to , which is the location of the minimum. ∎
Proposition 3**.**
If and with being number the of batches . (Each batch defines a parabola. The empirical loss is the mean of these parabolas). And for all it holds that and that is positive definite. Then holds.
Proof.
Since is a sum of convex functions, it is also convex and has one minimum.
At first we determine the derivative of with respect to :
[TABLE]
Then we determine the minima:
[TABLE]
[TABLE]
Thus, we get:
[TABLE]
∎
Appendix D Further experimental results
D.1 Performance Comparison on ImageNet, CIFAR-10, CIFAR-100 and Tolstoi
D.2 Wall-clock time comparison
D.3 SLS ResNet34 test case re-implementation
In the shown experiments and in contrast to the evaluation of SLS in [60], we used Tensorflow default Xavier weight initialization [19] versus PyTorch default Lecun initialization [33]. In addition, we used L2 regularisation versus no regularization. Furthermore, default implementations of networks for both frameworks have small differences. All in all, those differences usually influence the optimizer performance only marginally, as given by the fact that all other investigated optimizers perform well. However, in this case of SLS we see significant differences.
To prove that our implementation of SLS is correct, we re-implemented [60]’s ResNet34 test case on CIFAR-10 in Tensorflow and achieved similar results as [60]. SLS shows well performance and is not significantly overfitting as it does in in Section 5.2.
D.4 Parabolic property in adapted directions:
D.5 Influence of dynamic step sizes and the direction adaptation
This section analyses, whether PAL’s performance originates from dynamically chosen step sizes or from the the non-linear conjugate gradient like update step adaptation. We consider EfficientNets trained on CIFAR-10, since for those the update step adaptation factor is needed to achieve optimal results. We consider the following 6 scenarios: 1,2) PAL without update step adaptation () and with and without dynamic step sizes (Figure 18 left). 3,4) PAL with a update step adaptation of and with and without dynamic step sizes (Figure 18 middle). 5,6) PAL with a update step adaptation of with and without fixed step sizes (Figure 18 right). The case with fixed step sizes result in in normalized SGD (NSGD) with a momentum factor . As fixed update step size we use the measuring step size .
The results show that dynamic step sizes increase the performance always if direction adaptation is not applied and if it is applied in 6 out of 8 cases. Direction adaptation can increase or decrease the performance in both, the dynamic and the fixed step size cases. The best performance is achieved with a direction adaptation factor of and a measuring step size of , which shows that both factors influence the best results in this scenario.
D.6 Sensitivity analysis:
All in all PAL tends to have a low hyperparameter sensitivity as shown in Figure 19. Since is the most sensitive hyperparameter we analyzed its sensitivity over several further models trained on CIFAR-10 (see Figure 20).
D.7 Comparison to Probabilistic Line-Search (PLS):
We used a empirically improved and only existing implementation of PLS [38] for Tensorflow 1 [2]. However, the sum of squared gradients has to be derived manually for each layer, which is a considerable amount of work for modern architectures. Consequently, we limit our comparison to a ResNet-32 trained on CIFAR-10. Figure 21 shows that PAL and PLS perform similarly in this scenario. Default hyperparameters were used.
D.8 Further experimental design details
D.8.1 Training Procedure
On CIFAR-10 and CIFAR-100 we trained 150k steps. On Imagenet each network was trained for 500k steps. We performed a piecewise constant learning rate decay by dividing the learning rate by 10 at 50% and 75% of the steps.
The training set to evaluation set split was 45k to 15k for CIFAR-10 and CIFAR-100. At the time of writing, the default Tensorflow classes do not support the reuse of the same randomly sampled numbers for multiple inferences, therefore, we implemented and used our own Dropout [57] layer.
To get a fair comparison of the optimizers capabilities, we compare on the training loss, the validation accuracy and the test accuracy metrics. For all metrics we provide the median and the quartiles to analyze the hyperparameter sensitivity. For each hyperparameter combination we averaged our results over 3 runs using the seeds 1, 2 and 3 for reproducibility. All in all, we trained over 4500 networks with Tensorflow 1.15 [1] on Nvidia Geforce GTX 1080 TI graphic cards.
D.8.2 Data Augmentation
On CIFAR-10 we performed the following augmentations [23]:
4 pixel padding and cropping, horizontal image flipping with probability 0.5.
On Imagenet we applied an initial random crop to 224x224 pixels. In addition, we applied lighting as described in [32]. For CIFAR-10, CIFAR-100 all images were normalized by channel-wise mean and variance. For the Tolstoi War and Peace dataset we omitted augmentation.
D.8.3 Hyperparameter grid search
For our evaluation we used all combinations out of the following common used hyperparameters. The batch size is always 128 except for DenseNets trained with ALIG, SGDHD and COCOB for which we encountered memory overflows and had to reduce the batch size to 100. Weight decay is always .
On Imagenet, such a comprehensive grid search was not possible. In this case we compared with the best hyperparameter combinations found on Cifar-100.
ADAM:
hyperparameter symbol values
learning rate
first momentum
second momentum
epsilon
We did not vary the first or second momentum much, since [30] states that the values chosen are already good defaults.
SGD:
hyperparameter symbol values
learning rate
momentum
RMSProp:
hyperparameter symbol values
learning rate
discounting factor
epsilon
PAL:
hyperparameter symbol values
measuring step size
direction adaptation factor
update step adaptation
maximum step size
In our implementation we worked with a inverse update step adaptation .
SLS:
hyperparameter symbol values
initial step size
step size decay
step size reset
Armijo constant
maximum step size
ALIG:
hyperparameter symbol values
maximal learning rate
momentum
COCOB:
hyperparameter symbol values
restriction factor
SGDHD:
hyperparameter symbol values
learning rate
hyper gradient learning rate
D.9 Detailed numerical results
Appendix E Binary Line Search
The optimal binary line search we compared PAL against. Since the line decreases in negative gradient direction, at first a extrapolation phase performs as many steps forward as the loss does not increase. Afterwards a binary search is performed. This approach is valid if the underlying line is convex. For simple readability we chose Python 3.6 syntax.
1Input:max_num_of_search_steps
2def binary_line_search(last_loss, step, counter, is_extrapolate):
3 if counter == max_num_of_search_steps:
4 return last_loss
5 counter += 1
6 if is_extrapolate:
7 current_loss = do_step_on_line(step)
8 if current_loss < last_loss:
9 return binary_line_search(current_loss, step,counter, is_extrapolate)
10 else:
11 is_extrapolate = False
12 do_step_on_line(-step,get_loss=False)
13 if not is_extrapolate:
14 loss_right = do_step_on_line(0.5*step, True)
15 if loss_right < last_loss:
16 return binary_line_search(loss_right, 0.5*step,counter, is_extrapolate)
17 loss_left = do_step_on_line(-1*step, True)
18 if loss_left < last_loss:
19 return binary_line_search(loss_left, 0.5*step,counter, is_extrapolate)
20 do_step_on_line(0.5*step,get_loss=False)
21 if loss_right >= last_loss and loss_left >= last_loss:
22 return binary_line_search(loss_left, 0.5*step, counter, is_extrapolate)
23 else:
24 # this state is not possible
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor Flow: Large-scale machine lea
- 2[2] L. Balles. Probabilistic line search tensorflow implementation, 2017.
- 3[3] A. G. Baydin, R. Cornish, D. M. Rubio, M. Schmidt, and F. Wood. Online learning rate adaptation with hypergradient descent. ar Xiv preprint ar Xiv:1703.04782 , 2017.
- 4[4] A. S. Berahas, M. Jahani, and M. Takác. Quasi-newton methods for deep learning: Forget the past, just sample. Co RR , abs/1901.09997, 2019.
- 5[5] L. Berrada, A. Zisserman, and M. P. Kumar. Training neural networks for and by interpolation. ar Xiv preprint ar Xiv:1906.05661 , 2019.
- 6[6] N. Bostrom and E. Yudkowsky. The ethics of artificial intelligence. The Cambridge handbook of artificial intelligence , 1:316–334, 2014.
- 7[7] A. Botev, H. Ritter, and D. Barber. Practical gauss-newton optimisation for deep learning. ar Xiv preprint ar Xiv:1706.03662 , 2017.
- 8[8] A. Botev, H. Ritter, and D. Barber. Practical gauss-newton optimisation for deep learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 557–565. JMLR. org, 2017.