A Unifying View of Coverage in Linear Off-Policy Evaluation

TL;DR

This paper introduces a unified framework for understanding coverage in linear off-policy evaluation in reinforcement learning, providing a new coverage parameter that generalizes previous notions and offers tighter finite-sample guarantees.

Contribution

It proposes a novel coverage parameter called feature-dynamics coverage, unifying various existing definitions and enabling a comprehensive analysis of linear OPE algorithms.

Findings

Introduces feature-dynamics coverage parameter.

Provides finite-sample error bounds based on this new coverage.

Recovers classical coverage notions under additional assumptions.

Abstract

Off-policy evaluation (OPE) is a fundamental task in reinforcement learning (RL). In the classic setting of linear OPE, finite-sample guarantees often take the form $$ \textrm{Evaluation error} \le \textrm{poly}(C^\pi, d, 1/n,\log(1/\delta)), $$ where $d$ is the dimension of the features and $C^\pi$ is a coverage parameter that characterizes the degree to which the visited features lie in the span of the data distribution. While such guarantees are well-understood for several popular algorithms under stronger assumptions (e.g. Bellman completeness), the understanding is lacking and fragmented in the minimal setting where only the target value function is linearly realizable in the features. Despite recent interest in tight characterizations of the statistical rate in this setting, the right notion of coverage remains unclear, and candidate definitions from prior analyses have undesirable properties and are starkly disconnected from more standard definitions in the literature. We provide a novel finite-sample analysis of a canonical algorithm for this setting, LSTDQ. Inspired by an instrumental-variable view, we develop error bounds that depend on a novel coverage parameter, the feature-dynamics coverage, which can be interpreted as linear coverage in an induced dynamical system for feature evolution. With further assumptions -- such as Bellman-completeness -- our definition successfully recovers the coverage parameters specialized to those settings, finally yielding a unified understanding for coverage in linear OPE.