Multiscale replay: A robust algorithm for stochastic variational inequalities with a Markovian buffer

TL;DR

The paper presents Multiscale Experience Replay (MER), an algorithm that accelerates convergence in stochastic variational inequalities with Markovian data, without needing to know the chain's mixing time.

Contribution

MER introduces a multi-scale sampling scheme that overcomes bias in serial sampling, improving convergence speed in dependent data scenarios without prior mixing time knowledge.

Findings

MER achieves faster iteration complexity in VI problems.

MER is robust and does not require mixing time knowledge.

Applications include policy evaluation and dependent data modeling.

Abstract

We introduce the Multiscale Experience Replay (MER) algorithm for solving a class of stochastic variational inequalities (VIs) in settings where samples are generated from a Markov chain and we have access to a memory buffer to store them. Rather than uniformly sampling from the buffer, MER utilizes a multi-scale sampling scheme to emulate the behavior of VI algorithms designed for independent and identically distributed samples, overcoming bias in the de facto serial scheme and thereby accelerating convergence. Notably, unlike standard sample-skipping variants of serial algorithms, MER is robust in that it achieves this acceleration in iteration complexity whenever possible, and without requiring knowledge of the mixing time of the Markov chain. We also discuss applications of MER, particularly in policy evaluation with temporal difference learning and in training generalized linear models with dependent data.