Do optimization methods in deep learning applications matter?

TL;DR

This paper compares various optimization methods in deep learning, highlighting the trade-offs between convergence speed and computational complexity across different algorithms and applications.

Contribution

It provides an experimental comparison of first and higher-order optimization functions, analyzing their performance and computational costs in deep learning tasks.

Findings

Levenberg-Marquardt outperforms in convergence speed

LM has significantly higher training time

CG and SGD are more practical for large-scale applications

Abstract

With advances in deep learning, exponential data growth and increasing model complexity, developing efficient optimization methods are attracting much research attention. Several implementations favor the use of Conjugate Gradient (CG) and Stochastic Gradient Descent (SGD) as being practical and elegant solutions to achieve quick convergence, however, these optimization processes also present many limitations in learning across deep learning applications. Recent research is exploring higher-order optimization functions as better approaches, but these present very complex computational challenges for practical use. Comparing first and higher-order optimization functions, in this paper, our experiments reveal that Levemberg-Marquardt (LM) significantly supersedes optimal convergence but suffers from very large processing time increasing the training complexity of both, classification and reinforcement learning problems. Our experiments compare off-the-shelf optimization functions(CG, SGD, LM and L-BFGS) in standard CIFAR, MNIST, CartPole and FlappyBird experiments.The paper presents arguments on which optimization functions to use and further, which functions would benefit from parallelization efforts to improve pretraining time and learning rate convergence.