# Cocoercivity, Smoothness and Bias in Variance-Reduced Stochastic Gradient Methods

## TL;DR

This paper introduces SVAG, a variance-reduced stochastic gradient method with adjustable bias, analyzing its convergence under cocoercivity and smoothness assumptions, and examining the practical effects of bias in classification tasks.

## Contribution

It presents SVAG, a novel biased variance-reduced method, and provides convergence analysis highlighting the limitations of cocoercivity assumptions in biased settings.

## Key findings

- SVAG's step-size requirements are less restrictive when operators are gradients.
- Bias affects early convergence stages more than later stages.
- Theoretical convergence conditions match or improve previous results, but differ from practical observations.

## Abstract

With the purpose of examining biased updates in variance-reduced stochastic gradient methods, we introduce SVAG, a SAG/SAGA-like method with adjustable bias. SVAG is analyzed in a cocoercive root-finding setting, a setting which yields the same results as in the usual smooth convex optimization setting for the ordinary proximal-gradient method. We show that the same is not true for SVAG when biased updates are used. The step-size requirements for when the operators are gradients are significantly less restrictive compared to when they are not. This highlights the need to not rely solely on cocoercivity when analyzing variance-reduced methods meant for optimization. Our analysis either match or improve on previously known convergence conditions for SAG and SAGA. However, in the biased cases they still do not correspond well with practical experiences and we therefore examine the effect of bias numerically on a set of classification problems. The choice of bias seem to primarily affect the early stages of convergence and in most cases the differences vanish in the later stages of convergence. However, the effect of the bias choice is still significant in a couple of cases.