Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis

TL;DR

This paper presents an improved optimizer called Enhanced NIRMAL with damped Nesterov acceleration, demonstrating superior convergence stability and generalization on image classification benchmarks compared to several existing optimizers.

Contribution

The paper introduces Enhanced NIRMAL, integrating damped Nesterov acceleration into NIRMAL, and evaluates its performance against standard optimizers on multiple datasets.

Findings

Enhanced NIRMAL achieves higher accuracy on CIFAR-100 than original NIRMAL.

Enhanced NIRMAL shows competitive performance with SGD with Momentum.

The optimizer demonstrates improved stability and convergence on complex datasets.

Abstract

This study introduces the Enhanced NIRMAL (Novel Integrated Robust Multi-Adaptation Learning with Damped Nesterov Acceleration) optimizer, an improved version of the original NIRMAL optimizer. By incorporating an $(\alpha, r)$-damped Nesterov acceleration mechanism, Enhanced NIRMAL improves convergence stability while retaining chess-inspired strategies of gradient descent, momentum, stochastic perturbations, adaptive learning rates, and non-linear transformations. We evaluate Enhanced NIRMAL against Adam, SGD with Momentum, Nesterov, and the original NIRMAL on four benchmark image classification datasets: MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, using tailored convolutional neural network (CNN) architectures. Enhanced NIRMAL achieves a test accuracy of 46.06\% and the lowest test loss (1.960435) on CIFAR-100, surpassing the original NIRMAL (44.34\% accuracy) and closely rivaling SGD with Momentum (46.43\% accuracy). These results underscore Enhanced NIRMAL's superior generalization and stability, particularly on complex datasets.