# Extended Regularized Dual Averaging Methods for Stochastic Optimization

## TL;DR

This paper proposes the extended regularized dual averaging (XRDA) algorithm for stochastic optimization, offering flexible backward step size control to improve convergence while maintaining sparsity and theoretical guarantees.

## Contribution

It introduces XRDA, a novel extension of RDA that allows bounded backward step sizes, enhancing convergence and practical performance in stochastic optimization.

## Key findings

- XRDA achieves comparable convergence rates to RDA.
- Flexible step size control improves convergence speed.
- The method preserves sparsity in solutions.

## Abstract

We introduce a new algorithm, extended regularized dual averaging (XRDA), for solving regularized stochastic optimization problems, which generalizes the regularized dual averaging (RDA) method. The main novelty of the method is that it allows a flexible control of the backward step size. For instance, the backward step size used in RDA grows without bound, while for XRDA the backward step size can be kept bounded. We demonstrate experimentally that additional control over the backward step size can significantly improve the convergence rate of the algorithm while preserving desired properties of the iterates, such as sparsity. Theoretically, we show that the XRDA method achieves the same convergence rate as RDA for general convex objectives.