Beyond Worst-Case Analysis in Stochastic Approximation: Moment Estimation Improves Instance Complexity

TL;DR

This paper investigates how moment estimation enhances the efficiency of stochastic approximation algorithms by focusing on instance-dependent complexity rather than worst-case scenarios, bridging theory and practice.

Contribution

It introduces a new instance-dependent complexity bound and an adaptive algorithm that leverages moment estimation to achieve this bound without prior noise knowledge.

Findings

Proposes a new domination relation among complexity bounds.

Introduces an adaptive algorithm with moment estimation.

Provides theoretical justification for improved instance complexity.

Abstract

We study oracle complexity of gradient based methods for stochastic approximation problems. Though in many settings optimal algorithms and tight lower bounds are known for such problems, these optimal algorithms do not achieve the best performance when used in practice. We address this theory-practice gap by focusing on instance-dependent complexity instead of worst case complexity. In particular, we first summarize known instance-dependent complexity results and categorize them into three levels. We identify the domination relation between different levels and propose a fourth instance-dependent bound that dominates existing ones. We then provide a sufficient condition according to which an adaptive algorithm with moment estimation can achieve the proposed bound without knowledge of noise levels. Our proposed algorithm and its analysis provide a theoretical justification for the success of moment estimation as it achieves improved instance complexity.