Stochastic Difference-of-Convex Optimization with Momentum

TL;DR

This paper introduces a momentum-based stochastic difference-of-convex optimization method that guarantees convergence under standard assumptions for any batch size, addressing limitations of existing approaches.

Contribution

The work demonstrates that momentum enables convergence in stochastic DC optimization with small batches, a significant improvement over prior methods requiring large batches or strong noise assumptions.

Findings

Momentum ensures convergence under standard assumptions for any batch size.

Without momentum, convergence can fail regardless of stepsize.

Empirical results show strong performance of the proposed method.

Abstract

Stochastic difference-of-convex (DC) optimization is prevalent in numerous machine learning applications, yet its convergence properties under small batch sizes remain poorly understood. Existing methods typically require large batches or strong noise assumptions, which limit their practical use. In this work, we show that momentum enables convergence under standard smoothness and bounded variance assumptions (of the concave part) for any batch size. We prove that without momentum, convergence may fail regardless of stepsize, highlighting its necessity. Our momentum-based algorithm achieves provable convergence and demonstrates strong empirical performance.