Accelerating Optimization via Differentiable Stopping Time

TL;DR

This paper introduces a differentiable stopping time framework that allows for gradient-based optimization of the time to reach a target loss, enabling faster and more efficient hyperparameter tuning and optimization in machine learning.

Contribution

It proposes a novel differentiable stopping time based on differential equations, facilitating gradient-based optimization for timing in machine learning algorithms.

Findings

Superior performance in diverse experiments

Effective for online hyperparameter tuning

Enables differentiable optimization of stopping times

Abstract

Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.