A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning

TL;DR

This paper reviews various optimization strategies for gradient descent in deep learning, addressing issues of speed and accuracy, and discusses algorithm architectures and neural network optimization techniques.

Contribution

It provides a comprehensive overview of optimization methods for gradient descent and explores their architectures and enhancements for neural network training.

Findings

Different optimization algorithms improve convergence speed.

Optimization strategies enhance neural network performance.

Discussion of algorithm architectures aids in understanding improvements.

Abstract

One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value of a function we need gradient. And to update our weights we need gradient descent. But there are some problems with regular gradient descent ie. it is quite slow and not that accurate. This article aims to give an introduction to optimization strategies to gradient descent. In addition, we shall also discuss the architecture of these algorithms and further optimization of Neural Networks in general