Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

TL;DR

This paper enhances the stochastic gradient descent optimizer for deep learning by integrating numerical optimization techniques, demonstrating improved performance across various tasks and emphasizing the importance of optimization theory in neural network training.

Contribution

It introduces a novel SGD enhancement inspired by numerical methods, bridging optimization theory and deep learning for better interpretability and accuracy.

Findings

Improved training accuracy on multiple deep learning benchmarks.

Enhanced interpretability of the optimization process.

Demonstrated robustness across diverse tasks.

Abstract

Optimization theory serves as a pivotal scientific instrument for achieving optimal system performance, with its origins in economic applications to identify the best investment strategies for maximizing benefits. Over the centuries, from the geometric inquiries of ancient Greece to the calculus contributions by Newton and Leibniz, optimization theory has significantly advanced. The persistent work of scientists like Lagrange, Cauchy, and von Neumann has fortified its progress. The modern era has seen an unprecedented expansion of optimization theory applications, particularly with the growth of computer science, enabling more sophisticated computational practices and widespread utilization across engineering, decision analysis, and operations research. This paper delves into the profound relationship between optimization theory and deep learning, highlighting the omnipresence of optimization problems in the latter. We explore the gradient descent algorithm and its variants, which are the cornerstone of optimizing neural networks. The chapter introduces an enhancement to the SGD optimizer, drawing inspiration from numerical optimization methods, aiming to enhance interpretability and accuracy. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy. The paper concludes by emphasizing the continuous development of optimization theory and its expanding role in solving intricate problems, enhancing computational capabilities, and informing better policy decisions.