Gradient Descent Efficiency Index

TL;DR

This paper introduces the Ek efficiency index to evaluate gradient descent iterations, helping to identify optimal stopping points and improve training efficiency in resource-limited settings.

Contribution

The study proposes a novel efficiency metric, Ek, that measures iteration effectiveness by considering error change and stability, aiding in optimization and model comparison.

Findings

Ek correlates with convergence behavior across datasets

It helps identify optimal stopping points in training

The index enhances decision-making in resource-constrained environments

Abstract

Gradient descent is a widely used iterative algorithm for finding local minima in multivariate functions. However, the final iterations often either overshoot the minima or make minimal progress, making it challenging to determine an optimal stopping point. This study introduces a new efficiency metric, Ek, designed to quantify the effectiveness of each iteration. The proposed metric accounts for both the relative change in error and the stability of the loss function across iterations. This measure is particularly valuable in resource-constrained environments, where costs are closely tied to training time. Experimental validation across multiple datasets and models demonstrates that Ek provides valuable insights into the convergence behavior of gradient descent, complementing traditional performance metrics. The index has the potential to guide more informed decisions in the selection and tuning of optimization algorithms in machine learning applications and be used to compare the "effectiveness" of models relative to each other.