ZENITH: Automated Gradient Norm Informed Stochastic Optimization

TL;DR

ZENITH is a novel optimizer that adaptively adjusts learning rates based on gradient norm evolution, leading to improved accuracy and efficiency in computer vision tasks without additional overhead.

Contribution

The paper introduces ZENITH, a gradient norm-informed optimizer that automatically adapts learning rates, outperforming existing methods in accuracy, efficiency, and compatibility with regularization.

Findings

Higher test accuracy than baselines across CNN architectures.

Faster training times in image classification tasks.

Superior mAP in object detection, keypoint detection, and segmentation.

Abstract

Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.