Three Operator Splitting with Subgradients, Stochastic Gradients, and Adaptive Learning Rates

TL;DR

This paper extends Three Operator Splitting (TOS) to handle stochastic gradients, subgradients, and adaptive learning rates, enabling more practical and efficient optimization in machine learning applications with complex or unknown structures.

Contribution

The paper introduces three extensions of TOS: handling subgradients, stochastic gradients, and adaptive step-sizes, with convergence guarantees and practical advantages.

Findings

Extensions ensure $\\mathcal{O}(1/\sqrt{t})$ convergence rate.

AdapTOS adapts to unknown smoothness, achieving universal convergence.

Empirical results show improved performance over competing methods.

Abstract

Three Operator Splitting (TOS) (Davis & Yin, 2017) can minimize the sum of multiple convex functions effectively when an efficient gradient oracle or proximal operator is available for each term. This requirement often fails in machine learning applications: (i) instead of full gradients only stochastic gradients may be available; and (ii) instead of proximal operators, using subgradients to handle complex penalty functions may be more efficient and realistic. Motivated by these concerns, we analyze three potentially valuable extensions of TOS. The first two permit using subgradients and stochastic gradients, and are shown to ensure a $\mathcal{O}(1/\sqrt{t})$ convergence rate. The third extension AdapTOS endows TOS with adaptive step-sizes. For the important setting of optimizing a convex loss over the intersection of convex sets AdapTOS attains universal convergence rates, i.e., the rate adapts to the unknown smoothness degree of the objective. We compare our proposed methods with competing methods on various applications.