Revisiting Stochastic Proximal Point Methods: Generalized Smoothness and Similarity

TL;DR

This paper introduces a generalized phi-smoothness framework for stochastic proximal point methods, extending their theoretical understanding and practical applicability in nonsmooth optimization problems in machine learning.

Contribution

It proposes a novel phi-smoothness framework for SPPM, providing comprehensive analysis without traditional smoothness assumptions, and extends applicability under the expected similarity assumption.

Findings

Theoretical analysis encompasses existing results as special cases.

Validated the framework through practical experiments.

Extended SPPM applicability to broader scenarios.

Abstract

The growing prevalence of nonsmooth optimization problems in machine learning has spurred significant interest in generalized smoothness assumptions. Among these, the (L0, L1)-smoothness assumption has emerged as one of the most prominent. While proximal methods are well-suited and effective for nonsmooth problems in deterministic settings, their stochastic counterparts remain underexplored. This work focuses on the stochastic proximal point method (SPPM), valued for its stability and minimal hyperparameter tuning-advantages often missing in stochastic gradient descent (SGD). We propose a novel phi-smoothness framework and provide a comprehensive analysis of SPPM without relying on traditional smoothness assumptions. Our results are highly general, encompassing existing findings as special cases. Furthermore, we examine SPPM under the widely adopted expected similarity assumption, thereby extending its applicability to a broader range of scenarios. Our theoretical contributions are illustrated and validated by practical experiments.