Norm-Constrained Flows and Sign-Based Optimization: Theory and Algorithms

TL;DR

This paper provides a theoretical and algorithmic analysis of Sign Gradient Descent (SignGD), revealing its connection to norm-constrained flows, introducing new variants, and establishing convergence guarantees for strongly convex problems.

Contribution

It offers a continuous-time perspective on SignGD, introduces new algorithmic variants using Filippov's framework, and proves convergence in strongly convex settings.

Findings

SignGD is an Euler discretization of a norm-constrained gradient flow.

New algorithmic variants approximate Filippov solutions at discontinuities.

Proven convergence guarantees for SignGD in strongly convex optimization.

Abstract

Sign Gradient Descent (SignGD) is a simple yet robust optimization method, widely used in machine learning for its resilience to gradient noise and compatibility with low-precision computations. While its empirical performance is well established, its theoretical understanding remains limited. In this work, we revisit SignGD from a continuous-time perspective, showing that it arises as an Euler discretization of a norm-constrained gradient flow. This viewpoint reveals a trust-region interpretation and connects SignGD to a broader class of methods defined by different norm constraints, such as normalized gradient descent and greedy coordinate descent. We further study the discontinuous nature of the underlying dynamics using Filippov's differential inclusion framework, which allows us to derive new algorithmic variants, such as the convex-combination sliding update for the $\ell_1$-constrained flow, that faithfully approximate Filippov solutions even at discontinuity points. While we do not provide convergence guarantees for these variants, we demonstrate that they preserve descent properties and perform well empirically. We also introduce an accelerated version of SignGD based on a momentum-augmented discretization of the sign-gradient flow, and show its effectiveness in practice. Finally, we establish provable convergence guarantees for standard SignGD in the setting of strongly convex optimization. Our results provide new geometric, algorithmic, and analytical insights into SignGD and its norm-constrained extensions.