The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications

TL;DR

The paper introduces the Ball-Proximal Point Method (BPM), a novel optimization algorithm inspired by PPM, which achieves linear convergence in non-smooth, non-convex settings and broadens the scope of proximal methods.

Contribution

It proposes the BPM algorithm with a ball constraint, proving its linear convergence and global guarantees under weaker assumptions than classical methods.

Findings

BPM converges linearly in non-convex, non-smooth regimes.

BPM retains convergence guarantees under weaker assumptions.

BPM serves as a blueprint for developing practical optimization algorithms.

Abstract

Non-smooth and non-convex global optimization poses significant challenges across various applications, where standard gradient-based methods often struggle. We propose the Ball-Proximal Point Method, Broximal Point Method, or Ball Point Method (BPM) for short - a novel algorithmic framework inspired by the classical Proximal Point Method (PPM) (Rockafellar, 1976), which, as we show, sheds new light on several foundational optimization paradigms and phenomena, including non-convex and non-smooth optimization, acceleration, smoothing, adaptive stepsize selection, and trust-region methods. At the core of BPM lies the ball-proximal ("broximal") operator, which arises from the classical proximal operator by replacing the quadratic distance penalty by a ball constraint. Surprisingly, and in sharp contrast with the sublinear rate of PPM in the nonsmooth convex regime, we prove that BPM converges linearly and in a finite number of steps in the same regime. Furthermore, by introducing the concept of ball-convexity, we prove that BPM retains the same global convergence guarantees under weaker assumptions, making it a powerful tool for a broader class of potentially non-convex optimization problems. Just like PPM plays the role of a conceptual method inspiring the development of practically efficient algorithms and algorithmic elements, e.g., gradient descent, adaptive step sizes, acceleration (Ahn & Sra, 2020), and "W" in AdamW (Zhuang et al., 2022), we believe that BPM should be understood in the same manner: as a blueprint and inspiration for further development.