A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models

TL;DR

This paper provides a detailed theoretical analysis of Fitted Q-evaluation (FQE), establishing optimal convergence rates and error bounds under both parametric and nonparametric models, and explores the impact of horizon length and probability ratios.

Contribution

It offers the first comprehensive theoretical understanding of FQE estimators, including optimal rates and the influence of horizon and probability ratios under mild assumptions.

Findings

FQE can achieve the parametric rate of $n^{-1/2}$ under non-parametric models.

Error bounds depend on the horizon $T$, with improvements under certain assumptions.

Using ratio functions can improve error bounds from $T^{1.5}/\sqrt{n}$ to $T/\sqrt{n}$.

Abstract

In this paper, we delve into the statistical analysis of the fitted Q-evaluation (FQE) method, which focuses on estimating the value of a target policy using offline data generated by some behavior policy. We provide a comprehensive theoretical understanding of FQE estimators under both parameteric and nonparametric models on the $Q$-function. Specifically, we address three key questions related to FQE that remain largely unexplored in the current literature: (1) Is the optimal convergence rate for estimating the policy value regarding the sample size $n$ ($n^{-1/2}$) achievable for FQE under a non-parametric model with a fixed horizon ($T$)? (2) How does the error bound depend on the horizon $T$? (3) What is the role of the probability ratio function in improving the convergence of FQE estimators? Specifically, we show that under the completeness assumption of $Q$-functions, which is mild in the non-parametric setting, the estimation errors for policy value using both parametric and non-parametric FQE estimators can achieve an optimal rate in terms of $n$. The corresponding error bounds in terms of both $n$ and $T$ are also established. With an additional realizability assumption on ratio functions, the rate of estimation errors can be improved from $T^{1.5}/\sqrt{n}$ to $T/\sqrt{n}$, which matches the sharpest known bound in the current literature under the tabular setting.