Normalized Gradients for All

TL;DR

This paper introduces a method using normalized gradients to adapt to H"{o}lder smoothness in optimization, with bounds depending on a new concept of local smoothness, inspired by Levy [2017].

Contribution

It presents a black-box approach to adapt to H"{o}lder smoothness using normalized gradients and introduces a novel local smoothness measure.

Findings

Provides a new adaptive optimization technique.

Establishes bounds based on local H"{o}lder smoothness.

Builds on Levy's 2017 work for gradient normalization.

Abstract

In this short note, I show how to adapt to H\"{o}lder smoothness using normalized gradients in a black-box way. Moreover, the bound will depend on a novel notion of local H\"{o}lder smoothness. The main idea directly comes from Levy [2017].