# Parabolic Approximation Line Search for DNNs

## TL;DR

This paper introduces a parabolic approximation line search method for deep neural network optimization that adaptively finds effective step sizes by exploiting the local convexity of loss functions, improving over traditional methods.

## Contribution

It presents a novel line search technique based on parabolic approximation that automatically adjusts step sizes in DNN training, eliminating the need for hand-designed schedules.

## Key findings

- Outperforms other step size estimation methods in experiments.
- Competitively matches common optimization algorithms across various datasets.
- Shows that local loss landscapes are often convex and suitable for parabolic approximation.

## Abstract

A major challenge in current optimization research for deep learning is to automatically find optimal step sizes for each update step. The optimal step size is closely related to the shape of the loss in the update step direction. However, this shape has not yet been examined in detail. This work shows empirically that the batch loss over lines in negative gradient direction is mostly convex locally and well suited for one-dimensional parabolic approximations. By exploiting this parabolic property we introduce a simple and robust line search approach, which performs loss-shape dependent update steps. Our approach combines well-known methods such as parabolic approximation, line search and conjugate gradient, to perform efficiently. It surpasses other step size estimating methods and competes with common optimization methods on a large variety of experiments without the need of hand-designed step size schedules. Thus, it is of interest for objectives where step-size schedules are unknown or do not perform well. Our extensive evaluation includes multiple comprehensive hyperparameter grid searches on several datasets and architectures. Finally, we provide a general investigation of exact line searches in the context of batch losses and exact losses, including their relation to our line search approach.