Provably Efficient Reinforcement Learning via Surprise Bound

TL;DR

This paper introduces a provably efficient reinforcement learning algorithm that leverages a surprise bound to achieve near-optimal regret with fewer ERM problems, applicable to general value function approximation.

Contribution

The paper presents a new RL algorithm with theoretical guarantees that is computationally efficient and works with general value function classes satisfying Bellman-completeness.

Findings

Achieves $ ilde{O}( ext{poly}( ext{iota} H)\sqrt{T})$ regret bound.

Requires only $O(H \log K)$ ERM problems, significantly fewer than previous methods.

Effective in both linear and high-dimensional sparse linear settings.

Abstract

Value function approximation is important in modern reinforcement learning (RL) problems especially when the state space is (infinitely) large. Despite the importance and wide applicability of value function approximation, its theoretical understanding is still not as sophisticated as its empirical success, especially in the context of general function approximation. In this paper, we propose a provably efficient RL algorithm (both computationally and statistically) with general value function approximations. We show that if the value functions can be approximated by a function class that satisfies the Bellman-completeness assumption, our algorithm achieves an $\widetilde{O}(\text{poly}(\iota H)\sqrt{T})$ regret bound where $\iota$ is the product of the surprise bound and log-covering numbers, $H$ is the planning horizon, $K$ is the number of episodes and $T = HK$ is the total number of steps the agent interacts with the environment. Our algorithm achieves reasonable regret bounds when applied to both the linear setting and the sparse high-dimensional linear setting. Moreover, our algorithm only needs to solve $O(H\log K)$ empirical risk minimization (ERM) problems, which is far more efficient than previous algorithms that need to solve ERM problems for $\Omega(HK)$ times.

Equations

$$\pi =\{{\pi }_h{\}}_{h=1}^H:\mathcal{S}\times [H]\to \Delta (\mathcal{A})$$

$$(s_1,a_1,r_1),(s_2,a_2,r_2),\dots ,(s_H,a_H,r_H)$$

$$V_h^{\pi }(s)={\mathbb{E}}_{\pi }\left[\sum _{h^{\prime }=h}^H{r}_{h^{\prime }}|s_h=s\right],$$

$$Q_h^{\pi }(s,a)={\mathbb{E}}_{\pi }[\sum _{h^{\prime }=h}^H{r}_{h^{\prime }}|s_h=s,a_h=a].$$

$$\text{Reg}(K)=\sum _{k=1}^K\left(V_1^{\ast }(s_1^k)-V_1^{{\pi }^k}(s_1^k)\right),$$

$$w(\mathcal{F},s,a)=\underset{f,f^{\prime }\in \mathcal{F}}{\max }\left(f(s,a)-f^{\prime }(s,a)\right),$$

$$\Vert f{\Vert }_{\mathcal{Z}}=\sqrt{\sum _{(s,a)\in \mathcal{Z}}{f}^2(s,a)},$$

$$\Vert f{\Vert }_{\mathcal{D}}=\sqrt{\sum _{(s,a,r)\in \mathcal{D}}(f(s,a)-r{)}^2},$$

$$\Vert f{\Vert }_{\infty }=\underset{(s,a)\in \mathcal{S}\times \mathcal{A}}{\max }|f(s,a)|$$

$$f_V(\cdot ,\cdot )=r(\cdot ,\cdot )+\sum _{s^{\prime }\in \mathcal{S}}P(s^{\prime }|\cdot ,\cdot )V(s^{\prime }).$$

$$(f(s,a)-f^{\prime }(s,a){)}^2\le L_1{\mathbb{E}}_{s^{\prime }\sim {\mathcal{D}}_h(\pi )}{\mathbb{E}}_{a^{\prime }\sim {\pi }_h(s^{\prime })}\left[(f(s^{\prime },a^{\prime })-f^{\prime }(s^{\prime },a^{\prime }){)}^2\right]$$

$$L_1\le \underset{\pi ,h\in [H]}{\sup }\frac{1}{{\lambda }_{\min }\left({\mathbb{E}}_{s\sim {\mathcal{D}}_h(\pi ),a\sim {\pi }_h(s)}\left[\varphi (s,a)\varphi (s,a{)}^{\mathsf{T}}\right]\right)}.$$

$$L_1\le \underset{\pi ,h\in [H]}{\sup }\frac{4s}{{\psi }_{\min }\left({\mathbb{E}}_{s\sim {\mathcal{D}}_h(\pi ),a\sim {\pi }_h(s)}\left[\varphi (s,a)\varphi (s,a{)}^{\mathsf{T}}\right]\right)},$$

$$f_h^m\leftarrow \arg \underset{f\in \mathcal{F}}{\min }\Vert f{\Vert }_{{\mathcal{D}}_h^m}^2$$

$$Q_h^m(\cdot ,\cdot )=\min \left\{f_h^m(\cdot ,\cdot )+b_h^m(\cdot ,\cdot ),H\right\}.$$

$$\underset{f_1,f_2\in \mathcal{F}}{\max }{f}_1(s,a)-f_2(s,a)$$

$$\text{s.t.}\Vert f_1-f_2{\Vert }_{\mathcal{Z}}\le \epsilon .$$

$$O(\iota \cdot H^{3/2}\cdot \sqrt{T}),$$

$${\hat{\mathcal{F}}}_h^m=\left\{f\in \mathcal{F}|\Vert f-{\hat{f}}_h^m{\Vert }_{{\hat{Z}}^m}^2\le \beta \right\}.$$

$${\lambda }_{\min }\left({\mathbb{E}}_{s^{\prime }\sim {\mathcal{D}}_h(\pi )}{\mathbb{E}}_{a^{\prime }\sim {\pi }_h(s^{\prime })}\left[\varphi (s^{\prime },a^{\prime })\varphi (s^{\prime },a^{\prime }{)}^{\mathsf{T}}\right]\right)$$

$${\psi }_{\min }\left({\mathbb{E}}_{s^{\prime }\sim {\mathcal{D}}_h(\pi )}{\mathbb{E}}_{a^{\prime }\sim {\pi }_h(s^{\prime })}\left[\varphi (s^{\prime },a^{\prime })\varphi (s^{\prime },a^{\prime }{)}^{\mathsf{T}}\right]\right)$$

$${\Vert f_V(\cdot ,\cdot )-r(\cdot ,\cdot )+\sum _{s^{\prime }\in \mathcal{S}}P(s^{\prime }|\cdot ,\cdot )V(s^{\prime })\Vert }_{\infty }\le \zeta .$$

$$\beta =c^{\prime }(L_1{H}^2{\ln }^3(T/\delta )\ln (\mathcal{N}(\mathcal{F},\delta /T^3))\ln (\mathcal{N}(\mathcal{S}\times \mathcal{A},\delta /T^2))+HT\zeta ).$$

$$O(\iota \cdot H^{3/2}\cdot \sqrt{T}+\sqrt{L_1\cdot H^2\cdot \zeta \cdot \log T}\cdot T),$$

$$t\ge 4L_1^2\ln \frac{8\mathcal{N}(\mathcal{F},{\epsilon }_0{)}^2}{\delta }$$

$$X_z=\{\begin{array}{ll}1/p& \text{z is added into Z′ for 1/p times}\\ 0& \text{otherwise}\end{array}.$$

$$\mathrm{Pr}\{|{\mathcal{Z}}^{\prime }|>4|\mathcal{Z}|/\delta \}\le \delta /4$$

$$(1-\epsilon )\Vert f-f^{\prime }{\Vert }_{\mathcal{Z}}^2-2\lambda \le \Vert f-f^{\prime }{\Vert }_{{\mathcal{Z}}^{\prime }}^2\le (1+\epsilon )\Vert f-f^{\prime }{\Vert }_{\mathcal{Z}}^2+8|\mathcal{Z}|\lambda /\delta .$$

$$p\ge 384L_1\cdot \ln (4\mathcal{N}(\mathcal{F},{\epsilon }_0)/\delta )/({\epsilon }^2\cdot |\mathcal{Z}|).$$

$$(1-\epsilon )\Vert f-f^{\prime }{\Vert }_{\mathcal{Z}}^2-2\lambda \le \Vert f-f^{\prime }{\Vert }_{{\mathcal{Z}}^{\prime }}^2\le (1+\epsilon )\Vert f-f^{\prime }{\Vert }_{\mathcal{Z}}^2+8|\mathcal{Z}|\lambda /\delta .$$

$$g_k=\sum _{h=1}^H(f(s_h^k,a_h^k)-f^{\prime }(s_h^k,a_h^k){)}^2.$$

Full text

1 Introduction

Modern Reinforcement Learning (RL) problems are often challenging due to the huge state spaces, and in practice, value function approximation schemes are usually employed to tackle this issue. Empirically, combining various reinforcement learning algorithms with function approximation schemes has led to tremendous success on various tasks (Mnih et al., 2013, 2015; Silver et al., 2017). However, despite the great empirical success, our theoretical understanding of RL with function approximation is still not as sophisticated as its empirical counterpart. Until recently, most existing theoretical work in RL has been focusing on the tabular setting or the linear setting (Azar et al., 2017; Jin et al., 2018; Yang and Wang, 2019; Wang et al., 2019; Du et al., 2019b, a; Agarwal et al., 2020; Wang et al., 2020a; Du et al., 2020; Jin et al., 2020; Zanette et al., 2020; Li et al., 2020), while in practice, complex function approximators like neural networks are usually employed. Over the years, understanding conditions on the function class that permit sample-efficient RL has evolved into an important open research problem in machine learning theory.

Existing provably efficient RL algorithms that can handle general function approximation (Jiang et al., 2017; Sun et al., 2019; Ayoub et al., 2020; Jin et al., 2021; Du et al., 2021) usually require solving computationally intractable optimization problems and are therefore computationally inefficient. Recently, Wang et al. (2020b) proposed a provably efficient RL algorithm with general function approximation for function classes with bounded eluder dimensions. The algorithm by Wang et al. (2020b) is based on Least Squares Value Iteration (LSVI) and the principle of “optimism in the face of uncertainty”. There are two shortcomings in the work of Wang et al. (2020b). First, in order to calculate the exploration bonus, their algorithm applies sensitivity sampling (Langberg and Schulman, 2010; Feldman and Langberg, 2011; Feldman et al., 2013) to reduce the size of the replay buffer. Using a replay buffer with bounded complexity to calculate the exploration bonus is crucial for the correctness of their algorithm. On the other hand, such a step is complicated in nature and could be hard to implement in practice. Therefore, to make the algorithm practical, it is much more desirable to use simpler dimensionality reduction techniques (like uniform sampling) without sacrificing the theoretical guarantee. Second, as mentioned in Foster et al. (2018), showing examples with a small eluder dimension beyond linearly parameterized functions is challenging. In addition, taking the worst-case over all histories, as in the definition of the eluder dimension, is usually overly pessimistic in practice. In contextual bandits, it is known that provable efficiency can be established by assuming distributional conditions on the problem. For example, Foster et al. (2018) establishes regret bound for an optimism-based contextual bandits algorithm by assuming bounded surprise bound. It is natural to ask whether similar conditions can be used to establish provable efficiencies of RL algorithms.

Recently, Foster et al. (2020) established instance-dependent regret bounds for contextual bandits and reinforcement learning problems by assuming a bounded disagreement coefficient, which is a distribution-dependent assumption. Foster et al. (2020) show that the disagreement coefficient is always upper bounded by the eluder dimension of the function class. The RL algorithm in Foster et al. (2020), which is also based on Least Squares Value Iteration (LSVI) and the principle of “optimism in the face of uncertainty”, has two drawbacks. First, their algorithm achieves provable guarantees only in the block MDP setting which might not be realistic in practice. Second, when calculating the exploration bonus, their algorithm uses the star hull to reduce the complexity of the replay buffer, which is also complicated in nature and therefore difficult to implement in practice.

In this paper, we develop a novel provably efficient RL algorithm with general function approximation. Similar to previous algorithms (Wang et al., 2020b; Foster et al., 2020), our algorithm is an optimistic version of LSVI. Compared to previous ones, our algorithm has the following advantages:

• •

The regret bound of our algorithm is based on a variant of surprise bound proposed in (Foster et al., 2018), which is a distribution-dependent quantity and could therefore be smaller than the eluder dimension which considers the worst-case over all histories. Moreover, our theory does not rely on the block MDP assumption. Furthermore, the surprise bound can be upper bounded in the tabular setting, the linear setting and the high dimensional sparse linear setting, which implies our algorithm achieves reasonable regret bound in all these three settings.

• •

The dimensionality reduction technique for reducing the complexity of the replay buffer is based on uniform sampling. This is much simpler than the sensitivity sampling framework in Wang et al. (2020b) and the method based on star hull in Foster et al. (2020).

• •

Our algorithm requires solving only $O(H\log K)$ empirical risk minimization (ERM) problems, while previous algorithms (Wang et al., 2020b; Foster et al., 2020) require solving $\Omega(HK)$ ERM problems.

1.1 Related work

Tabular reinforcement learning.

Tabular RL is well studied in the context of sample complexity and regret bound in numerous literature (Kearns and Singh, 2002; Kakade, 2003; Strehl et al., 2006, 2009; Jaksch et al., 2010; Azar et al., 2013; Lattimore and Hutter, 2014; Dann and Brunskill, 2015; Agrawal and Jia, 2017; Azar et al., 2017; Jin et al., 2018; Dann et al., 2019; Zanette and Brunskill, 2019; Zhang et al., 2020; Wang et al., 2020a; Yang et al., 2021). In particular, for episodic MDP without further assumptions, the best regret bound is $\widetilde{O}(\sqrt{H^{2}SAT})$ for both model-based (Azar et al., 2017) and model-free (Zhang et al., 2020) algorithms, which matches the lower bound $\Omega\left(\sqrt{H^{2}SAT}\right)$ proved by Jin et al. (2018). Recently, Yang et al. (2021) propose an RL algorithm with a regret bound of $O\left(\frac{SA\text{poly}(H)}{\Delta_{\min}}\log(SAT)\right)$ assuming the existence of a positive sub-optimality gap. However, all algorithms mentioned above cannot be applied to RL problems with huge or infinite state spaces due to the polynomial dependence on $\sqrt{S}$ in the regret bound. Therefore, in this paper, we assume the value function lies in a function class with bounded complexity and design a provably efficient algorithm whose regret bound depends polynomially on the complexity of the function class instead of the size of the state space.

Bandits.

There is also rich literature studying stochastic (contextual) bandits, which can be viewed as a special case of MDP without state transitions (Auer, 2002; Dani et al., 2008; Li et al., 2010; Rusmevichientong and Tsitsiklis, 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Foster et al., 2018, 2020; Li et al., 2019). In particular, Foster et al. (2018) study contextual bandit problems with general value function approximation, and prove their algorithms could achieve a regret bound depending polynomially on the surprise bound and the implicit exploration coefficient (IEC). In this paper, we study RL with general value function approximation, and prove that the regret bound of our algorithm also depends on the (slightly modified) surprise bound as well as the log-covering numbers. However, we note that the RL setting is much more complicated than the contextual bandits setting since there is no state transition in bandit problems.

Reinforcement learning with function approximation.

In the setting of linear function approximation, there has been great interest recently in the theoretical analysis of the sample complexity of RL algorithms (Yang and Wang, 2019, 2020; Jin et al., 2020; Cai et al., 2020; Du et al., 2019b, 2020; Wang et al., 2019; Zanette et al., 2020; Zhou et al., 2021). Compared to linear function approximation, however, many current provably efficient algorithms for general value function approximation are relatively impractical. For example, algorithms in Jiang et al. (2017); Sun et al. (2019); Dong et al. (2020) achieve regret bound in terms of the witness rank or the Bellman rank, but they are not computationally efficient. Foster et al. (2020) devise RegRL algorithm which is both computationally and statistically efficient. However, it requires the block MDP assumption which greatly alleviates the difficulty of (infinitely) huge state space and might not be realistic in practice. Ayoub et al. (2020) propose a model-based algorithm and Wang et al. (2020b) propose a model-free algorithm for general value function approximation, and the regret bound of both algorithms depend on the eluder dimension. Kong et al. (2021) propose an efficient algorithm both computationally and statistically for general value function approximation, of which the regret bound also depends on the eluder dimension. However, the eluder dimension considers the worst-case over all histories and is thus often overly pessimistic. Instead, the regret bound of our algorithm depends polynomially on the surprise bound which is a distribution-dependent quantity and thus could be smaller than the eluder dimension for practical scenarios.

2 Preliminaries

In this paper, we study episodic Markov Decision Process (MDP) $\mathcal{M}=(\mathcal{S},\mathcal{A},H,P,r,\mu)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the finite action space, $H\in\mathbb{N}_{+}$ is the planning horizon, $P:\mathcal{S}\times\mathcal{A}\ \to\Delta(\mathcal{S})$ is the transition kernel which maps a state-action pair to a distribution over the state space, $r:\mathcal{S}\times\mathcal{A}\to[0,1]$ is the reward function and $\mu\in\Delta(\mathcal{S})$ is the initial state distribution 111Our analysis can be naturally extended to the time-inhomogeneous settings where the reward function and the transition kernel are different for each $h\in[H]$..

A (stochastic) policy

[TABLE]

maps any state $s$ to a distribution over the action space at each step $h$, where we use $[N]$ to denote the set $\{1,2,\ldots,N\}$ for any positive integer $N$. A trajectory

[TABLE]

is induced by a policy $\pi$ if $s_{1}\sim\mu$, $a_{h}\sim\pi_{h}(s_{h}),r_{h}=r(s_{h},a_{h}),\forall h\in[H]$ and $s_{h+1}\sim P(s_{h},a_{h}),\forall h\in[H-1]$. Furthermore, a policy $\pi=\{\pi_{h}\}_{h=1}^{H}$ is deterministic if for each step $h\in[H]$, $\pi_{h}:\mathcal{S}\to\mathcal{A}$ maps a state to only one action.

For any policy $\pi$, the expected cumulative reward starting from state $s$ at step $h$ is defined as the value function

[TABLE]

where we use superscript $\pi$ to denote that the trajectory is induced by $\pi$. Similarly, the expected cumulative reward starting from state-action pair $(s,a)$ at step $h$ is defined as the $Q$-function

[TABLE]

Let $\pi^{*}$ denote the optimal policy which maximizes $\mathbb{E}_{s_{1}\sim\mu}[V_{1}^{\pi}(s_{1})]$. Also, let $V_{h}^{*}(s)=V_{h}^{\pi^{*}}(s)$ and $Q_{h}^{*}(s,a)=Q_{h}^{\pi^{*}}(s,a)$.

The agent interacts with the environment for $K$ episodes. At the beginning of each episode $k\in[K]$, the agent specifies a policy $\pi^{k}$ based on previous trajectories and interacts with the environment using $\pi^{k}$ for $H$ steps. We assume the agent knows the number of episodes $K$, and we define $T=KH$ to be the total number of steps that the agent interacts with the environment. The regret of an algorithm after $K$ episodes is defined as

[TABLE]

which compares the accumulated rewards between the agent’s policy and the optimal policy. The goal of the agent is to minimize the regret. In this paper, we consider the typical regime that $H$ is fixed while $K$ grows to infinity.

Width function and norms.

For notation convenience, we define the width function for any function class $\mathcal{F}\subseteq\{f:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\}$ and several norms for any function $f:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$. The width function is defined as

[TABLE]

$\forall(s,a)\in\mathcal{S}\times\mathcal{A}$. For any dataset $\mathcal{Z}\subseteq\mathcal{S}\times\mathcal{A}$ and $\mathcal{D}\subseteq\mathcal{S}\times\mathcal{A}\times\mathbb{R}$, define $\mathcal{Z}$-norm

[TABLE]

$\mathcal{D}$-norm

[TABLE]

and infinite norm

[TABLE]

respectively. In addition, define $\|v\|_{\infty}=\max_{s\in\mathcal{S}}|v(s)|$ for any $v:\mathcal{S}\to\mathbb{R}$.

Additional notations for algorithms.

For any finite multiset $\mathcal{X}$, let $\texttt{Unif}(\mathcal{X})$ denote the uniform distribution over $\mathcal{X}$ and $\text{Card}_{\text{d}}(\mathcal{X})$ denote the number of distinct elements in $\mathcal{X}$. For any $x\in\mathbb{R}_{+}$, let $\lfloor x\rfloor$ denote the integer part of $x$ and define $\lceil x\rceil=\lfloor x\rfloor+1$ if $x$ is not an integer and otherwise $\lceil x\rceil=x$. We use the standard $O(\cdot),\Omega(\cdot)$ notations to hide constants and use $\tilde{O}(\cdot),\tilde{\Omega}(\cdot)$ to suppress log factors. Also, we use $x\lesssim y$ to denote that there exists a constant $c>0$ s.t. $x\leq cy$, and use $x\gtrsim y$ if $y\lesssim x$.

3 Algorithm

In this section, we first introduce the assumptions for the algorithm and then present our main algorithm (Algorithm 1). The theoretical guarantee of our algorithm is presented in Section 4.

3.1 Assumptions

Assume our algorithm (Algorithm 1) receives a function class $\mathcal{F}\subseteq\{f:\mathcal{S}\times\mathcal{A}\to[0,H+1]\}$ as part of the input. Since the complexity of $\mathcal{F}$ determines the efficiency of the algorithm, it is natural and necessary to require bounded complexities of the function class under appropriate measures. We make the following assumptions on the function class $\mathcal{F}$.

Assumption 1** (Bellman-completeness).**

For any function $V:\mathcal{S}\to[0,H]$, there exists a function $f_{V}\in\mathcal{F}$, s.t.

[TABLE]

Assumption 1 indicates the closedness under Bellman equations. This is a general assumption that summarizes many previous assumptions in special settings and is commonly adopted in previous literature for general value function approximation (Wang et al., 2020b; Foster et al., 2020; Kong et al., 2021). For tabular RL, $\mathcal{F}$ can be chosen as the set of all functions mapping from $\mathcal{S}\times\mathcal{A}$ to $[0,H+1]$. In the linear MDP setting (Bradtke and Barto, 1996; Jin et al., 2020; Yang and Wang, 2019, 2020; Wang et al., 2019) where the transition kernel and the reward function are both linear in a feature map $\phi:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{d}$, $\mathcal{F}$ can be the set of all linear functions with respect to $\phi$. In sparse high-dimensional linear MDP settings where the transition kernel and the reward function are both $s$-sparse linear functions in $\phi$, $\mathcal{F}$ can be the set of all $(2s)$-sparse linear functions with respect to $\phi$. Furthermore, Assumption 1 approximately holds in practice as long as $\mathcal{F}$ is rich enough (e.g., deep neural networks) and we show in Section 5 that our algorithm is robust to model misspecification.

Assumption 2** (Bounded covering number).**

Given any $\varepsilon>0$, there exist covering sets $\mathcal{C}(\mathcal{F},\varepsilon)\subseteq\mathcal{F}$ and $\mathcal{C}(\mathcal{S}\times\mathcal{A},\varepsilon)\subseteq\mathcal{S}\times\mathcal{A}$ with bounded size $\mathcal{N}(\mathcal{F},\varepsilon)$ and $\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)$ respectively, where

• •

$\forall f\in\mathcal{F}$, $\exists f^{\prime}\in\mathcal{C}(\mathcal{F},\varepsilon),$ s.t. $\|f-f^{\prime}\|_{\infty}\leq\varepsilon$.

• •

$\forall(s,a)\in\mathcal{S}\times\mathcal{A}$, $\exists(s^{\prime},a^{\prime})\in\mathcal{C}(\mathcal{S}\times\mathcal{A},\varepsilon),$ s.t. $\max_{f\in\mathcal{F}}|f(s,a)-f(s^{\prime},a^{\prime})|\leq\varepsilon$.

Assumption 2 requires bounded covering numbers $\mathcal{N}(\cdot,\varepsilon)$ for both $\mathcal{F}$ and $\mathcal{S}\times\mathcal{A}$, and the regret bound of our algorithm depends only logarithmically on the covering numbers (Theorem 1). In the tabular RL setting, $\ln\mathcal{N}(\mathcal{F},\varepsilon)=\widetilde{O}(|\mathcal{S}||\mathcal{A}|)$ and $\ln\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)=O(\ln(|\mathcal{S}||\mathcal{A}|))$. In $d$-dimensional linear MDP settings, $\ln\mathcal{N}(\mathcal{F},\varepsilon)=\widetilde{O}(d)$ and $\ln\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)=\widetilde{O}(d)$. In $s$-sparse high-dimensional linear MDP settings, $\ln\mathcal{N}(\mathcal{F},\varepsilon)=\widetilde{O}(s)$. If we further assume that $\phi(s,a)$ is $s$-sparse for all $(s,a)\in\mathcal{S}\times\mathcal{A}$, then $\ln\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)=\widetilde{O}(s)$.

Surprise bound.

Another important complexity measure in this paper is surprise bound, which was first introduced in Foster et al. (2018) to characterize the complexity of the function class in the contextual bandit setting.

Definition 1** (Surprise bound).**

The surprise bound is the smallest positive constant $L_{1}$ s.t.

[TABLE]

for all $f,f^{\prime}\in\mathcal{F},s\in\mathcal{S},a\in\mathcal{A},h\in[H]$ and any policy $\pi$, where $\mathcal{D}_{h}(\pi)$ is the distribution of $s_{h}$ when the policy is $\pi$.

Intuitively, the surprise bound is small if all pairs of functions with a small expected squared error with respect to any policy, do not encounter a much larger squared error on any state-action pair. The following proposition gives upper bounds of the surprise bound for linear and sparse linear settings (see Appendix C for the proof).

Proposition 3.1**.**

In the (sparse) linear MDP setting with a fixed feature map $\phi:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^{d}$, consider the function class $\mathcal{F}=\{(s,a)\mapsto w^{\mathsf{T}}\phi(s,a)|w\in\mathcal{W}\}$ for some $\mathcal{W}\subseteq\mathbb{R}^{d}$.

• •

If $\|\phi(s,a)\|_{2}\leq 1,\forall(s,a)\in\mathcal{S}\times\mathcal{A}$ and $\|w\|_{2}\leq 2H\sqrt{d},\forall w\in\mathcal{W}$, then

[TABLE]

• •

If $\|\phi(s,a)\|_{\infty}\leq 1,\forall(s,a)\in\mathcal{S}\times\mathcal{A}$ and $\|w\|_{\infty}\leq 2H\sqrt{d},\|w\|_{0}\leq 2s,\forall w\in\mathcal{W}$, then

[TABLE]

where $\psi_{\min}(A)=\min_{w\neq 0:\|w\|_{0}\leq 4s}w^{\mathsf{T}}Aw/w^{\mathsf{T}}w$ is the minimum restricted eigenvalue for $(4s)$-sparse predictors (Raskutti et al., 2010).

3.2 Algorithm

In this section, we present our main algorithm (Algorithm 1) and discuss in detail several important components of our algorithm.

3.2.1 Doubling epoch schedule

Our algorithm consists of $M$ epochs where each epoch $m\in[M]$ starts at the beginning of episode $\tau_{m}=2^{m-1}$ and consists of $T_{m}=2^{m-1}$ episodes. Thus, the total number of episodes $K=2^{M}-1$ and $M=O(\log K)$. At the beginning of epoch $m$, the algorithm fixes a policy $\pi^{m}=\{\pi^{m}_{h}\}_{h=1}^{H}$ and the agent executes $\pi^{m}$ for all episodes $k\in[\tau_{m},\tau_{m}+T_{m}-1]$. The $M$ epochs can be divided into two phases.

• •

Phase 1: Warm-up epochs. For the first $(M_{0}-1)$ epochs, the agent plays a uniformly random policy. These warm-up epochs are designed to encourage exploration at the initial episodes.

• •

Phase 2: Optimistic LSVI. Starting from epoch $M_{0}$, we use an optimistic version of Least Squares Value Iteration (LSVI) similar to Jin et al. (2020); Wang et al. (2019, 2020b); Foster et al. (2020). At the beginning of each epoch $m\geq M_{0}$, we maintain all previous trajectories as a replay buffer, and find the best fit $f^{m}=\{f^{m}_{h}\}_{h=1}^{H}\in\mathcal{F}^{H}$ with respect to the replay buffer in the sense of mean squared error (MSE), i.e.,

[TABLE]

where $\mathcal{D}_{h}^{m}$ is the replay buffer (see definition in Algorithm 1). To avoid overfitting and encourage exploration, we design a bonus function $b_{h}^{m}(\cdot,\cdot)$ which we will discuss later in Section 3.2.2, and approximate the optimal $Q$ function $Q^{*}_{h}(\cdot,\cdot)$ by

[TABLE]

Our design of the bonus function ensures that $Q_{h}^{m}$ is an optimistic estimator of $Q_{h}^{*}$ with high probability (Lemma 6). Finally, for each episode $k\in[\tau_{m},\tau_{m+1}-1]$ in epoch $m$, the agent plays the greedy policy with respect to $Q_{h}^{m}$ and collect the trajectory in episode $k$.

The advantages of the doubling epoch schedule are two folded:

• •

Computationally efficient. Since our algorithm only conducts large amount of computation at the beginning of each epoch (computing $f^{m}_{h}$ by empirical risk minimization and $b^{m}_{h}$ by the width function as in Section 3.2.2, which can often be solved efficiently by appropriate optimization methods or assuming access to appropriate regression oracles (Wang et al., 2020b; Foster et al., 2018)) and there are only $O(\log K)$ epochs, our algorithm is much more computationally efficient than previous methods (Wang et al., 2020b; Foster et al., 2020) which require to solve $\Omega(HK)$ equivalent optimization problems.

Recently, Kong et al. (2021) proposes an online sub-sampling technique which improves the computational complexity of Wang et al. (2020b). However, our algorithm is still much more computationally efficient than Kong et al. (2021). The algorithm of Kong et al. (2021) adopts sensitivity sampling, which requires computing sensitivities for each state action pair $(s_{h}^{k},a_{h}^{k})$. Since the calculation of sensitivity requires solving a regression oracle for $\Omega(\log(TH))$ times (see Section 4.4. in Kong et al. (2021)), and there are $T=KH$ such state-action pairs, their algorithm needs to solve $\Omega(KH\log(TH))$ regression oracles to calculate sensitivities and subsample the dataset. While in our algorithm, we use uniform sampling to avoid the complex and time-consuming sensitivity calculation and thus does not need any oracle to perform the subsampling procedure.

• •

Stabilizing adjacent trajectories. The doubling epoch schedule together with the warm-up epochs stabilizes the adjacent trajectories by ensuring that at the beginning of each epoch, at least half of the historical trajectories in the replay buffer are induced by the same policy. This property enables us to adopt uniform sampling (Algorithm 2) to reduce the complexity of the replay buffer.

3.2.2 Uniform sampling

An important technical novelty of our algorithm is the design of the bonus function via uniform sampling. To ensure optimism of our estimator $Q_{h}^{m}$, we can choose $b_{h}^{m}$ as the upper bound of the difference between $Q_{h}^{*}$ and $f_{h}^{m}$. If we are able to obtain a confidence region $\mathcal{F}_{h}^{m}$ which contains both $f_{h}^{m}$ and $Q_{h}^{*}$, it suffices to define the bonus function as the width function of $\mathcal{F}_{h}^{m}$.

A naive way to choose the confidence region is $\mathcal{F}_{h}^{m}=\left\{f\in\mathcal{F}\left|\|f-f_{h}^{m}\|_{\mathcal{Z}^{m}}^{2}\leq\beta\right.\right\}$ with a carefully selected $\beta$. However, since the confidence region depends on the whole replay buffer with size at most $T$, the confidence region and thus the bonus function would suffer extremely high complexity. This implies that $\beta$ needs to be set extremely large to ensure the accuracy of the confidence region. To obtain a bonus function with low complexity, we reduce the complexity of the replay buffer by uniform sampling, which is formally stated in Algorithm 2.

Comparison to previous methods.

Actually, the algorithms in Wang et al. (2020b); Foster et al. (2020) also suffer the high complexity of the bonus function and address the issue by sensitivity sampling and star hull respectively. However, sensitivity sampling requires estimating the sensitivity of each state-action pair, which is time-consuming; the star hull is complicated in nature and thus is hard to implement in practice. In contrast, our uniform sampling is conceptually simple and easy to implement. Note that there is only one single parameter $p$ to be determined in Algorithm 2. When the surprise bound $L_{1}$ is known in advance, we can directly calculate the value of $p$. When $L_{1}$ is unknown, we can perform a grid-search in a log-space of $L_{1}$. Specifically, we can set a small value $L_{\min}$ as the lower bound of $L_{1}$ and a large value $L_{\max}$ as the upper bound, and perform Algorithm 1 for $L_{1}\in\mathcal{L}\triangleq\{L_{\min},2L_{\min},2^{2}L_{\min},\ldots,L_{\max}\}$. Then we can pick the policy with the best performance under different choices of $L_{1}$.

Theorem 1 shows that the regret of our main algorithm (Algorithm 1) is $\tilde{O}(\sqrt{T})$ in $T$ dependence. We also emphasize that the above grid-search procedure won’t result in higher total regret, since one can first try each possible $L_{1}\in\mathcal{L}$ for $O(\sqrt{T})$ times, and then exploit the best $L_{1}$ for the remaining $O(T-\sqrt{T}\log(L_{\max}/L_{\min}))=O(T)$ steps. The resulting total regret is still $\tilde{O}(\sqrt{T})$.

Design of the bonus function via uniform sampling.

Now we are able to design a bonus function with low complexity as in Algorithm 3 via uniform sampling. After obtaining the reduced dataset $\mathcal{Z}^{\prime}$, we round each data in $\mathcal{Z}^{\prime}$ and the reference function $\bar{f}$ to their nearest neighbors in covering sets. The confidence region and the bonus function is then defined by the rounded reference function and the rounded dataset. Note that in Algorithm 3, the rounding operation does not need to be performed explicitly since all the data are stored in computers with bounded precision, and thus all the data will be implicitly rounded. For the choice of $\beta$, we can use the same grid-search method of $L_{1}$ since $\beta$ is also determined by $L_{1}$.

Efficient computation of the bonus function.

The computation of the bonus function is equivalent to an optimization problem of the following form:

[TABLE]

This problem can be solved efficiently by either assuming access to an optimization oracle, or assuming access to only a regression oracle (which is a milder assumption than optimization oracles) as mentioned in Section 4.4 of Kong et al. (2021).

4 Theoretical results

In this section, we formally present our main theorem of the regret bound and defer the proof to Appendix B.

Theorem 1** (Main theorem).**

Under Assumption 1, 2, let $M_{0}=\left\lceil\ln\left(16L_{1}^{2}\ln\frac{128T\mathcal{N}(\mathcal{F},\delta/(9216T^{2}))^{2}}{\delta}\right)\right\rceil$ where the number of total steps $T=H\cdot(2^{M}-1)$ is sufficiently large. With probability at least $1-\delta$, the regret of Algorithm 1 is at most

[TABLE]

where $\iota=L_{1}\cdot\ln^{2}(T/\delta)\cdot\max(\ln(\mathcal{N}(\mathcal{F},\delta/T^{3})),\ln(\mathcal{N}(\mathcal{S}\times\mathcal{A},\delta/T^{2}))).$

Proof sketch.

In this proof sketch, we ignore the rounding operation in Algorithm 3 for convenience. The proof can be decomposed into three main steps.

• •

Step 1: Bounding the complexity of the bonus function. First, we show that our bonus function has low complexity (Proposition A.2). Note that the bonus function is defined as the width function of the confidence region

[TABLE]

Since the reduced dataset $\hat{\mathcal{Z}}^{m}$ has bounded size (Lemma 1) and bounded number of distinct elements (Lemma 3), our bonus function which is defined by $\hat{\mathcal{Z}}^{m}$ also has low complexity. Now it remains to show that the bonus function defined over the reduced dataset $\hat{\mathcal{Z}}^{m}$ is (almost) the same as the bonus function defined over the original dataset $\mathcal{Z}^{m}$. It is equivalent to show that the confidence region remains (almost) unchanged after uniform sampling. This can be proved by showing that for any function pairs $f,f^{\prime}\in\mathcal{F}$, the $\mathcal{Z}^{\prime}$-norm of $f-f^{\prime}$ approximates well the $\mathcal{Z}$-norm of $f-f^{\prime}$ (Lemma 2). For a fixed function pair $(f,f^{\prime})$, $\|f-f^{\prime}\|_{\mathcal{Z}^{\prime}}^{2}$ is an unbiased estimator of $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}$ and its variance can be controlled, since the trajectories in the replay buffer are stabilized by the doubling epoch and thus $\mathcal{Z}^{\prime}$ has low complexity after uniform sampling. Then we can apply the Bernstein inequality to a fixed function pair $(f,f^{\prime})$ to show that $\|f-f^{\prime}\|_{\mathcal{Z}^{\prime}}^{2}$ is close to $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}$ with high probability. Applying a union bound over all function pairs in the covering set of $\mathcal{F}$, we can obtain the desired result.

• •

Step 2: Optimism of the estimated $Q$-function. The next step is to show that the estimated $Q$-function is an optimistic version of the true $Q$-function of the optimal policy (Lemma 6). To achieve this, we need to show that the best fit $f_{h}^{m}$ is close to $r(\cdot,\cdot)+\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|\cdot,\cdot)V_{h+1}^{m}(s^{\prime})$. If $f_{h}^{m}$ and $V_{h+1}^{m}$ are independent, a standard concentration argument concludes the result. However, $V_{h+1}^{m}$ and $f_{h}^{m}$ are subtly dependent since they are both determined by the previous dataset. To address the difficulty, we first apply the standard concentration result on a fixed $V$ (Lemma 4), and then apply a union bound over all $V$ in a covering set (Lemma 5) to obtain the result. This method is similar to Wang et al. (2020b).

• •

Step 3: Regret decomposition. Finally, we decompose the regret by the summation of the bonus functions (Lemma 7). Then, we use similar arguments as in Foster et al. (2018) to bound each bonus term by the surprise bound separately since the bonus function is defined as the (approximate) width function of the confidence region.

∎

Remark 1**.**

Recently, Foster et al. (2021) proposes a high-level algorithm E2D. When applying E2D algorithm to our settings, one can show that it also achieves a similar regret bound $\tilde{O}(\operatorname{poly}(L_{1})\sqrt{T})$ (other parameters omitted). However, we want to emphasize that E2D algorithm is too high-level to implement in practice. The implementation of E2D algorithm requires an online estimation oracle (see Algorithm 1 in Foster et al. (2021)), which is a very strong assumption in RL settings. While in our algorithm, we only require a ERM oracle and a regression oracle, which are mild and common assumptions in machine learning problems.

While our algorithm works for general value function class, it also achieves reasonable regret in special cases.

Tabular settings.

In the tabular RL setting, it holds that $\ln\mathcal{N}(\mathcal{F},\varepsilon)=\widetilde{O}(|\mathcal{S}||\mathcal{A}|)$ and $\ln\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)=O(\ln(|\mathcal{S}||\mathcal{A}|))$. When $\mu(s)\geq\varepsilon$ and $P(s^{\prime}|s,a)\geq\varepsilon$ for all $s,s^{\prime}\in\mathcal{S}$, $a\in\mathcal{A}$ for a (not too) small positive value $\varepsilon$, $L_{1}=O(\text{poly}(|\mathcal{S}||\mathcal{A}|))$, which implies that the regret bound is $\tilde{O}(\text{poly}(|\mathcal{S}||\mathcal{A}|)H^{3/2}\sqrt{T})$. This is a reasonable regret bound since it is optimal in terms of $T$, the most important term in the regret bound, and has polynomial dependency in other parameters.

Linear settings.

When $\mathcal{F}$ is a $d$-dimensional linear function class, we have $\ln\mathcal{N}(\mathcal{F},\varepsilon)=\ln\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)=\widetilde{O}(d)$. When

[TABLE]

is lower bounded (of order $\Omega(1/d)$) and thus $L_{1}=O(d)$ by Proposition 3.1, the regret bound is $\widetilde{O}(d^{2}\cdot~{}H^{3/2}~{}\cdot\sqrt{T})$, which is optimal in $T$-dependency and matches the result of Wang et al. (2020b) in $d$-dependency.

Sparse linear settings.

Furthermore, when $\mathcal{F}$ is an $s$-sparse high-dimensional linear function class where typically $d\geq T\gg s$ , we have $\ln\mathcal{N}(\mathcal{F},\varepsilon)=\widetilde{O}(s)$. When

[TABLE]

is lower bounded (of order $\Omega(1)$) and thus $L_{1}$ is $O(s)$ by Proposition 3.1, the regret bound is $\widetilde{O}(s\cdot\max(s,\ln(\mathcal{N}(\mathcal{S}\times\mathcal{A},\delta/T^{2})))\cdot H^{3/2}\cdot\sqrt{T})$. If we further assume that $\phi(s^{\prime},a^{\prime})$ is $s$-sparse for all $(s^{\prime},a^{\prime})\in\mathcal{S}\times\mathcal{A}$, we have $\ln\mathcal{N}(\mathcal{S}\times\mathcal{A},\varepsilon)=\widetilde{O}(s)$ and thus obtain an $\widetilde{O}(s^{2}\cdot H^{3/2}\cdot\sqrt{T})$ regret bound. However, directly applying the result in linear settings of Wang et al. (2020b) can only obtain a linear regret when $d\geq T$. This shows the superiority of our algorithm since we can provide theoretical guarantee for more general function classes, and thus it is an important step toward studying general value function approximation beyond the tabular and linear settings.

We also emphasize a subtle difference between linear and sparse linear settings. In linear settings, when $\lambda_{\min}\left(\mathbb{E}_{s^{\prime}\sim\mathcal{D}_{h}(\pi)}\mathbb{E}_{a^{\prime}\sim\pi_{h}(s^{\prime})}\left[\phi(s^{\prime},a^{\prime})\phi(s^{\prime},a^{\prime})^{\mathsf{T}}\right]\right)$ is lower bounded, we typically expect it to be of order $\Omega(1/d)$ since we assume the 2-norm $\|\phi\|_{2}\leq 1$. While for sparse linear settings, when $\psi_{\min}\left(\mathbb{E}_{s^{\prime}\sim\mathcal{D}_{h}(\pi)}\mathbb{E}_{a^{\prime}\sim\pi_{h}(s^{\prime})}\left[\phi(s^{\prime},a^{\prime})\phi(s^{\prime},a^{\prime})^{\mathsf{T}}\right]\right)$ is lower bounded, we typically expect it to be of order $\Omega(1)$ since we assume the infinity norm $\|\phi\|_{\infty}\leq 1$ in this setting.

5 Model Misspecification

Our main theorem (Theorem 1) requires Bellman-completeness assumption (Assumption 1). Although the Bellman-completeness assumption is fairly common in theoretical analysis, especially in the presence of general value function approximation, the ground truth model together with the function class might slightly violate this assumption in real-world scenario. This phenomenon is known as model misspecification (Jin et al., 2020; Wang et al., 2020b).

In this section, we show that as long as the violation of the Bellman-completeness assumption is small, the regret of our algorithm is still bounded. To state the result formally, we first introduce the following assumption, which can be viewed as a model misspecification version of the Bellman-completeness assumption.

Assumption 3** (Model misspecification).**

There exists a constant $\zeta>0$ satisfying that for any function $V:\mathcal{S}\to[0,H]$, there exists a function $f_{V}\in\mathcal{F}$, s.t.

[TABLE]

Under Assumption 3, one can directly apply Algorithm 1 to the model misspecification setting with only a different choice of the parameter $\beta$ in Algorithm 3. Specifically, for some constant $c^{\prime}>0$ we set

[TABLE]

Note that when Assumption 1 holds, it is equivalent to Assumption 3 with $\zeta=0$, and thus the parameter $\beta$ is exactly the same as the one in our original algorithm. The following theorem provides theoretical guarantees of our algorithm for model misspecification, and the proof is attached in Appendix D, which is very similar to the proof of Theorem 1.

Theorem 2** (Theoretical guarantee for model misspecification).**

Under Assumption 3, 2, let $M_{0}=\left\lceil\ln\left(16L_{1}^{2}\ln\frac{128T\mathcal{N}(\mathcal{F},\delta/(9216T^{2}))^{2}}{\delta}\right)\right\rceil$ and the number of total steps $T=H\cdot(2^{M}-1)$. With probability at least $1-\delta$, the regret of Algorithm 1 (where the parameter $\beta$ is defined as in (1)) is at most

[TABLE]

where $\iota=L_{1}\cdot\ln^{2}(T/\delta)\cdot\max(\ln(\mathcal{N}(\mathcal{F},\delta/T^{3})),\ln(\mathcal{N}(\mathcal{S}\times\mathcal{A},\delta/T^{2}))).$

6 Conclusion

In this paper, we propose a provably efficient RL algorithm (both computationally and statistically) with general value function approximation. The regret bound of our algorithm depends on the surprise bound, which is a distribution-dependent quantity and could therefore be smaller than the eluder dimension considered in previous work. Our algorithm achieves reasonable regret bound when instantiating to special function classes.

As a future direction, it would be interesting to see if it is possible to establish the provable efficiency of RL algorithms using other distribution-dependent complexity measures. For example, it would be interesting to study whether it is possible to design a provably efficient RL algorithm by assuming a bounded disagreement coefficient (as in Foster et al. (2020)) but without the block MDP assumption.

Appendix A Analysis of the bonus function

In this section, we analyze our bonus function, and the main proposition is presented in Proposition A.2.

A.1 Analysis of Algorithm 2

Note that the notation $\delta$ in Algorithm 3 and Algorithm 2 are different. In this subsection, all the notation $\delta$ refer to $\delta$ in Algorithm 2, and therefore, $\lambda=\delta/(16T)$. Also, let $\varepsilon_{0}=\varepsilon/72\cdot\sqrt{\lambda\delta/|\mathcal{Z}|}$ throughout this subsection.

We assume that the input dataset of Algorithm 2 is $\mathcal{Z}=\{(s_{h}^{k},a_{h}^{k})\}_{(h,k)\in[H]\times[t]}$ where more than half of the trajectories are induced by the same policy and the number of trajectories

[TABLE]

which is satisfied if $t\geq\tau_{M_{0}}$ and $M_{0}$ is chosen as in Theorem 1.

The first lemma gives an upper bound on the size of the dataset produced by uniform sampling.

Lemma 1**.**

With probability at least $1-\delta/4$, $|\mathcal{Z}^{\prime}|\leq 4|\mathcal{Z}|/\delta$.

Proof.

We define random variable

[TABLE]

Since $|\mathcal{Z}^{\prime}|=\sum_{z\in\mathcal{Z}}X_{z}$ and $\mathbb{E}[X_{z}]=1$, we can obtain

[TABLE]

by Markov inequality. ∎

The next lemma proves that after uniform sampling, the norms of difference of any function pairs are approximately preserved with high probability.

Lemma 2**.**

With probability at least $1-\delta/2$, for any $f,f^{\prime}\in\mathcal{F}$,

[TABLE]

Proof.

When $p=1$, $\mathcal{Z}=\mathcal{Z}^{\prime}$, the result directly holds. So we only consider the case when $p<1$, which means

[TABLE]

We separately consider the cases when $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}<2\lambda$ and $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}\geq 2\lambda$.

For any function pair $f,f^{\prime}\in\mathcal{F}$ where $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}<2\lambda$, conditioned on the event in Lemma 1 which holds with probability at least $1-\delta/4$, we can obtain that $\|f-f^{\prime}\|_{\mathcal{Z}^{\prime}}^{2}\leq|\mathcal{Z}^{\prime}|\|f-f^{\prime}\|_{\mathcal{Z}}^{2}\leq 4|\mathcal{Z}|/\delta\cdot\|f-f^{\prime}\|_{\mathcal{Z}}^{2}\leq 8|\mathcal{Z}|\lambda/\delta.$ Also, by the fact that $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}<2\lambda$ and $\|f-f^{\prime}\|_{\mathcal{Z}^{\prime}}^{2}\geq 0$, we can conclude that

[TABLE]

In the remaining part of the proof, we consider the case that $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}\geq 2\lambda$.

We first fix any pair of distinct functions $f,f^{\prime}\in\mathcal{C}(\mathcal{F},\varepsilon_{0})$. Assume the first $u=\lfloor(t+1)/2\rfloor$ trajectories are all induced by the same policy $\pi$. Also, for any $1\leq k\leq u$, let

[TABLE]

Therefore,

[TABLE]

Note that

[TABLE]

Also, by Definition 1,

[TABLE]

Therefore, by Hoeffding’s inequality,

[TABLE]

Setting $v=\frac{1}{2}$, we can obtain

[TABLE]

Let $\mathcal{E}_{1}$ denote the event that

[TABLE]

then $\Pr\{\mathcal{E}_{1}\}\geq 1-\frac{\delta}{8\mathcal{N}(\mathcal{F},\varepsilon_{0})^{2}}$.

Now, we condition on $\mathcal{E}_{1}$ for the following analysis. For each $z\in\mathcal{Z}$, define

[TABLE]

Obviously, $\|f-f^{\prime}\|_{\mathcal{Z}^{\prime}}^{2}=\sum_{z\in\mathcal{Z}}X_{z},$ and $\mathbb{E}[X_{z}]=\left(f(z)-f^{\prime}(z)\right)^{2}$. Also,

[TABLE]

Moreover,

[TABLE]

Then, by Azuma-Bernstein’s Inequality,

[TABLE]

Since the above inequality holds conditioned on $\mathcal{E}_{1}$, if we do not condition on $\mathcal{E}_{1}$,

[TABLE]

By union bound, the inequality above implies that with probability at least $1-\delta/4$, for any $f,f^{\prime}\in\mathcal{C}(\mathcal{F},\varepsilon_{0})$,

[TABLE]

Denote the event above and the event in Lemma 1 by $\mathcal{E}_{2}$, where

[TABLE]

Now we condition on $\mathcal{E}_{2}$ where $\Pr\{\mathcal{E}_{2}\}\geq 1-\delta/2$. For any function pair $f,f^{\prime}\in\mathcal{F}$ where $\|f-f^{\prime}\|_{\mathcal{Z}}^{2}\geq 2\lambda$, there exists $\hat{f},\hat{f^{\prime}}\in\mathcal{C}(\mathcal{F},\varepsilon_{0})$, s.t.

[TABLE]

Therefore,

[TABLE]

by $\mathcal{E}_{2}$. Then we can obtain that

[TABLE]

By similar methods, we can also obtain that

[TABLE]

∎

We also give the bound of the number of distinct elements in $\mathcal{Z}^{\prime}$.

Lemma 3**.**

With probability at least $1-\delta/4$, $\text{Card}_{\text{d}}(\mathcal{Z}^{\prime})\leq 2304L_{1}\cdot\ln(4\mathcal{N}(\mathcal{F},\varepsilon_{0})/\delta)/\varepsilon^{2}.$

Proof.

First, note that

[TABLE]

since for any $0<x<1$, there must exists $\hat{x}\in[x,2x]$ s.t. $1/\hat{x}$ is an integer.

When $p=1$, which means $\mathcal{Z}=\mathcal{Z}^{\prime}$ and

[TABLE]

we have

[TABLE]

When $p<1$, we have $p\geq 384L_{1}\cdot\ln(4\mathcal{N}(\mathcal{F},\varepsilon_{0})/\delta)/(\varepsilon^{2}\cdot|\mathcal{Z}|)$. Now, For each $z\in\mathcal{Z}$, define

[TABLE]

Then the number of distinct elements in $\mathcal{Z}^{\prime}$ is upper bounded by $\sum_{z\in\mathcal{Z}}X_{z}$. Since $\mathbb{E}[X_{z}]=p$,

[TABLE]

By Chernoff bound,

[TABLE]

∎

A.2 Analysis of Algorithm 3

In this subsection, all the notation $\delta$ refer to $\delta$ in Algorithm 3. In other words, we replace all the $\delta$ in Section A.1 by $\delta/(16T)$. Also, we still assume that the input dataset of Algorithm 3 is $\mathcal{Z}=\{(s_{h}^{k},a_{h}^{k})\}_{(h,k)\in[H]\times[t]}$ where more than half of the trajectories are induced by the same policy and the number of trajectories $t$ satisfies

[TABLE]

which is satisfied if $t\geq\tau_{M_{0}}$ and $M_{0}$ is chosen as in Theorem 1.

Combining the three lemmas in Section A.1 with a union bound, we can obtain the following proposition.

Proposition A.1**.**

Let $\mathcal{Z}^{\prime}$ denote the dataset returned by Algorithm 2. With probability at least $1-\delta/(16T)$, $|Z^{\prime}|\leq 64T^{2}/\delta$, the number of distinct elements in $\mathcal{Z}^{\prime}$ does not exceed

[TABLE]

and for any $f,f^{\prime}\in\mathcal{F}$,

[TABLE]

By Proposition A.1, we can deduce the following proposition.

Proposition A.2**.**

For Algorithm 3, the following holds.

With probability at least $1-\delta/(16T)$,

[TABLE]

where $\underline{\mathcal{F}}=\left\{f\in\mathcal{F}\ |\ \|f-\bar{f}\|_{\mathcal{Z}}^{2}\leq\beta(\mathcal{F},\delta)\right\}$ and $\overline{\mathcal{F}}=\left\{f\in\mathcal{F}\ |\ \|f-\bar{f}\|_{\mathcal{Z}}^{2}\leq 12\beta(\mathcal{F},\delta)+12\right\}$. 2. 2.

There exists a function set $\mathcal{W}$ s.t. $\hat{w}(\cdot,\cdot)\in\mathcal{W}$ and

[TABLE]

for some absolute constant $C>0$ when $T$ is sufficiently large.

Proof.

For the first part, we condition on the event defined in Proposition A.1. We only need to prove that $\underline{\mathcal{F}}\subseteq\widehat{\mathcal{F}}\subseteq\overline{\mathcal{F}}$, where $\widehat{\mathcal{F}}$ is defined in Algorithm 3. For any $f\in\mathcal{F}$, we have

[TABLE]

Therefore,

[TABLE]

This means for any $f\in\underline{\mathcal{F}}$, we have $\|f-\bar{f}\|_{\mathcal{Z}}^{2}\leq\beta(\mathcal{F},\delta)$, which implies $\|f-\hat{f}\|_{\widehat{\mathcal{Z}}}^{2}\leq 3\beta(\mathcal{F},\delta)+2$, i.e., $f\in\widehat{\mathcal{F}}$. Similarly,

[TABLE]

So for any $f\in\widehat{\mathcal{F}}$, we have $\|f-\hat{f}\|_{\widehat{\mathcal{Z}}}^{2}\leq 3\beta(\mathcal{F},\delta)+2$, which implies $\|f-\bar{f}\|_{\mathcal{Z}}^{2}\leq 12\beta(\mathcal{F},\delta)+12$, i.e., $f\in\overline{\mathcal{F}}$.

For the second part, since function $\hat{w}(\cdot,\cdot)$ is uniquely defined by $\widehat{\mathcal{F}}$, we only need to analyze the maximal number of different possible function classes $\widehat{\mathcal{F}}$. When $|\mathcal{Z}^{\prime}|>64T^{2}/\delta$ or the number of distinct elements in $\mathcal{Z}^{\prime}$ is larger than

[TABLE]

$|\mathcal{Z}^{\prime}|=0$ and thus $\widehat{\mathcal{F}}=\mathcal{F}$. Otherwise, $\widehat{\mathcal{F}}$ is determined by $\widehat{\mathcal{Z}}$ and $\hat{f}$. Since $\hat{f}\in\mathcal{C}(\mathcal{F},1/(8\sqrt{64T^{2}/\delta}))$, the number of different $\hat{f}$ does not exceed $\mathcal{N}(\mathcal{F},1/(8\sqrt{64T^{2}/\delta}))$. Moreover, since there are at most

[TABLE]

distinct elements in $\widehat{\mathcal{Z}}$, where $|\widehat{\mathcal{Z}}|\leq 64T^{2}/\delta$ and each element belongs to $\mathcal{C}(\mathcal{S}\times\mathcal{A},1/(8\sqrt{64T^{2}/\delta}))$, the number of different $\widehat{\mathcal{Z}}$ is upper bounded by

[TABLE]

∎

Appendix B Analysis of the main algorithm

Now we start to prove the regret bound of Algorithm 1. The following lemma provides a bound on the estimation of a single backup.

Lemma 4** (Single step optimization error).**

Consider a fixed epoch $m\in[M]\backslash[M_{0}]$. We define

[TABLE]

as in Algorithm 1. Also, for any function $V:\mathcal{S}\to[0,H]$, we define

[TABLE]

and

[TABLE]

Then, for any function $V:\mathcal{S}\to[0,H]$ and $\delta\in(0,1)$, there exists an event $\mathcal{E}_{V,\delta}$ where $\Pr\{\mathcal{E}_{V,\delta}\}\geq 1-\delta$, s.t. conditioned on $\mathcal{E}_{V,\delta}$, for any $V^{\prime}:\mathcal{S}\to[0,H]$ with $\|V-V^{\prime}\|_{\infty}\leq 1/T$, we have

[TABLE]

for some constant $c^{\prime}>0$.

Proof.

For any $V:\mathcal{S}\to[0,H]$, we define

[TABLE]

and now we consider a fixed $V$. For any $f\in\mathcal{F}$, define

[TABLE]

Also, for any $(h,k)\in[H]\times[\tau_{m}-1]$, define $\mathbb{F}_{h}^{k}$ as the filtration induced by

[TABLE]

Then we have $\mathbb{E}[\xi_{h}^{k}(f)|\mathbb{F}_{h}^{k}]=0$ and $\mathbb{E}[(\xi_{h}^{k}(f))^{2}|\mathbb{F}_{h}^{k}]\leq 4(H+1)^{2}(f(s_{h}^{k},a_{h}^{k})-f_{V}(s_{h}^{k},a_{h}^{k}))^{2}$. Applying Lemma 10 of Kirschner and Krause [2018] by setting $\{X_{t}\}=\{\xi_{h}^{k}(f)\}$, we can obtain that with probability at least $1-\delta$,

[TABLE]

Applying a union bound of $\xi_{h}^{k}(f),-\xi_{h}^{k}(f)$ over all $f\in\mathcal{C}(\mathcal{F},1/T)$, we can further obtain that with probability at least $1-\delta$,

[TABLE]

holds for all $f\in\mathcal{C}(\mathcal{F},1/T)$.

Let $\mathcal{E}_{V,\delta}$ denote the above event, and for the rest of the proof, we condition on $\mathcal{E}_{V,\delta}$.

Now, for any $f\in\mathcal{F}$, there exists a function $g\in\mathcal{C}(\mathcal{F},1/T)$, s.t. $\|f-g\|_{\infty}\leq 1/T$. Therefore,

[TABLE]

For any $V^{\prime}:\mathcal{S}\to[0,H]$ with $\|V^{\prime}-V\|_{\infty}\leq 1/T$, we can obtain that

[TABLE]

Furthermore, for any $f\in\mathcal{F}$,

[TABLE]

If we let $f=\hat{f}_{V^{\prime}}$, since $\hat{f}_{V^{\prime}}=\arg\min_{f\in\mathcal{F}}\|f\|_{\mathcal{D}_{V^{\prime}}^{m}}$, we have

[TABLE]

which implies

[TABLE]

for some constant $c^{\prime}>0$. ∎

Lemma 5** (Confidence region).**

In Algorithm 1, for $m>M_{0}$, define confidence region

[TABLE]

Then with probability at least $1-\delta/16$, for all $(h,m)\in[H]\times([M]\backslash[M_{0}])$,

[TABLE]

given

[TABLE]

for some constant $c^{\prime}>0$. Here, $\mathcal{W}$ is given in Proposition A.2.

Proof.

By Proposition A.2, $b_{h}^{m}(\cdot,\cdot)\in\mathcal{W},\forall(h,m)\in[H]\times([M]\backslash[M_{0}]).$ Note that

[TABLE]

is a $(1/T)$-cover of

[TABLE]

i.e., there exists $q\in\mathcal{Q}$, s.t. $\|q-Q_{h+1}^{m}\|_{\infty}\leq 1/T$. Therefore,

[TABLE]

is a $(1/T)$-cover of $V_{h+1}^{m}$ with $\ln|\mathcal{V}|\leq\ln|\mathcal{W}|+\ln\mathcal{N}(\mathcal{F},1/T)+1$.

Now, for each $V\in\mathcal{V}$, let $\mathcal{E}_{V,\delta/(16|\mathcal{V}|T)}$ denote the event defined in Lemma 4. By union bound, $\Pr\{\bigcap_{V\in\mathcal{V}}\mathcal{E}_{V,\delta/(16|\mathcal{V}|T)}\}\geq 1-\delta/(16T).$ In the rest of the proof, we condition on the event $\bigcap_{V\in\mathcal{V}}\mathcal{E}_{V,\delta/(16|\mathcal{V}|T)}$.

Since $f_{h}^{m}=\arg\min_{f\in\mathcal{F}}\|f\|_{\mathcal{D}_{h}^{m}}^{2}$, and there exists $V\in\mathcal{V}$ s.t. $\|V-V_{h+1}^{m}\|_{\infty}\leq 1/T$, by Lemma 4, we have

[TABLE]

for some constant $c^{\prime}>0$. Applying a union bound over all $(h,m)\in[H]\times([M]\backslash[M_{0}])$, we have that with probability at least $1-\delta/16$,

[TABLE]

∎

The above lemma proves that the confidence region contains $r(\cdot,\cdot)+\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|\cdot,\cdot)V_{h+1}^{m}(s^{\prime})$ with high probability, which implies that all the estimated $Q$-function $Q_{h}^{m}$ are optimistic with high probability as well. We formally state the conclusion in the next lemma.

Lemma 6** (Optimistic $Q$-function).**

With probability at least $1-\delta/8$,

[TABLE]

for all $(h,m)\in[H]\times([M]\backslash[M_{0}])$ and $(s,a)\in\mathcal{S}\times\mathcal{A}$.

Proof.

Let $\mathcal{F}_{h}^{m}$ be the confidence region as defined in Lemma 5. Let $\mathcal{E}_{1}$ denote the event that

[TABLE]

By Lemma 5, $\Pr\{\mathcal{E}_{1}\}\geq 1-\delta/16$. Let $\mathcal{E}_{2}$ denote the event that

[TABLE]

By Proposition A.2 and union bound over all $(h,m)\in[H]\times([M]\backslash[M_{0}])$, $\Pr\{\mathcal{E}_{2}\}\geq 1-\delta/16$. We condition on $\mathcal{E}_{1}\cap\mathcal{E}_{2}$ in the rest of the proof, which holds with failure probability at most $\delta/8$.

By the definition of width function,

[TABLE]

Since $r(\cdot,\cdot)+\sum_{s^{\prime}\in\mathcal{S}}P(s^{\prime}|\cdot,\cdot)V_{h+1}^{m}(s^{\prime})\in\mathcal{F}_{h}^{m}$, we have

[TABLE]

Therefore, for all $(s,a)\in\mathcal{S}\times\mathcal{A},$

[TABLE]

Next, we start to prove $Q_{h}^{*}(\cdot,\cdot)\leq Q_{h}^{m}(\cdot,\cdot)$ by induction on $h$. When $h=H+1$, the inequality directly holds since $Q_{H+1}^{*}(\cdot,\cdot)=Q_{H+1}^{m}(\cdot,\cdot)=0$. Now for any $h\in[H]$, assume $Q_{h+1}^{*}(\cdot,\cdot)\leq Q_{h+1}^{m}(\cdot,\cdot)$. This also implies $V_{h+1}^{*}(\cdot)\leq V_{h+1}^{m}(\cdot)$. Therefore, for any $(s,a)\in\mathcal{S}\times\mathcal{A}$,

[TABLE]

which completes the proof. ∎

Now, we can decompose the regret and bound it by the summation of bonus functions.

Lemma 7** (Regret decomposition).**

With probability at least $1-\delta/4$,

[TABLE]

Proof.

For any step $h\in[H]$, epoch $m\in[M]\backslash[M_{0}]$ and episode $k$ in epoch $m$, define

[TABLE]

and define $\mathbb{F}_{h}^{k}$ as the filtration induced by

[TABLE]

Then $\mathbb{E}[\xi_{h}^{k}|\mathbb{F}_{h}^{k}]=0$ and $|\xi_{h}^{k}|\leq 2H$. By Azuma-Hoeffding inequality, with probability at least $1-\delta/8$,

[TABLE]

We condition on both this event and the event defined in Lemma 6 which also holds with probability at least $1-\delta/8$ in the rest of the proof.

Let $\pi^{0}$ denote the uniformly random policy adopted in the first $(M_{0}-1)$ epochs. By Lemma 6,

[TABLE]

For each $k$ and corresponding $m$, we have

[TABLE]

Therefore,

[TABLE]

∎

To prove the main theorem, we also need the next lemma.

Lemma 8**.**

With probability at least $1-\delta/2$, for all $(h,m)\in[H]\times([M]\backslash[M_{0}])$ and any $f,f^{\prime}\in\mathcal{F}$,

[TABLE]

Proof.

We first fix any $(h,m)\in[H]\times([M]\backslash[M_{0}])$. Define dataset

[TABLE]

Now we fix any pair of distinct functions $f,f^{\prime}\in\mathcal{C}(\mathcal{F},1/T)$. Also, for any episode $k\in[\tau_{m-1},~{}\tau_{m}~{}-~{}1~{}]~{}$, let

[TABLE]

Therefore,

[TABLE]

Note that

[TABLE]

Also, by Definition 1,

[TABLE]

Therefore, by Hoeffding’s inequality,

[TABLE]

Since

[TABLE]

by setting $v=\frac{1}{2}$, we can obtain that

[TABLE]

By a union bound over all such function pairs $(f,f^{\prime})$, this implies that with probaiblity at least $1-\delta/(2T)$, for any $f,f^{\prime}\in\mathcal{C}(\mathcal{F},1/T)$,

[TABLE]

Now we condition on the event above in the following part of the proof.

To simplify the notation, we denote

[TABLE]

For any pair of functions $f,f^{\prime}\in\mathcal{F}$, there exists $\hat{f},\hat{f}^{\prime}\in\mathcal{C}(\mathcal{F},1/T)$, s.t. $\|f-\hat{f}\|_{\infty}\leq 1/T$ and $\|f^{\prime}-\hat{f}^{\prime}\|_{\infty}\leq 1/T$. When $\|f-f^{\prime}\|_{\pi_{h}^{m-1}}^{2}\leq 64/T_{m-1}$, we can directly obtain that

[TABLE]

So we only consider the case when $\|f-f^{\prime}\|_{\pi_{h}^{m-1}}^{2}\geq 64/T_{m-1}$. Then, we have

[TABLE]

Therefore,

[TABLE]

which means

[TABLE]

Finally, we complete the proof by directly applying a union bound over all $(h,m)\in[H]\times([M]\backslash[M_{0}])$.

∎

Now we are ready to prove the main theorem.

Proof of Theorem 1.

We condition on the event defined in Lemma 5, Lemma 6, Lemma 7 and Lemma 8. Also, we condition on the event in Proposition A.2 after applying a union bound over all $(h,m)\in[H]\times([M]\backslash[M_{0}])$. With probability at least $1-\delta$, all the above events hold.

By Lemma 7, we have

[TABLE]

For any $(h,m)\in[H]\times([M]\backslash[M_{0}])$, we define

[TABLE]

where

[TABLE]

as defined in Algorithm 1. Let

[TABLE]

By Proposition A.2, $b_{h}^{m}(\cdot,\cdot)\leq w(\overline{\mathcal{F}}_{h}^{m},\cdot,\cdot)$. Then, for any episode $k\in[\tau_{m},\tau_{m+1}-1]$,

[TABLE]

Therefore,

[TABLE]

which implies

[TABLE]

Then, we can obtain that

[TABLE]

∎

Appendix C Proof of Proposition 3.1

In this section, we provide the proof of Proposition 3.1.

Proof of Proposition 3.1.

For linear settings, let $\mathcal{W}^{*}=\{w-w^{\prime}|w,w^{\prime}\in\mathcal{W}\}$, then by Definition 1,

[TABLE]

For sparse high-dimensional linear settings, let $\mathcal{W}^{*}=\{w-w^{\prime}|w,w^{\prime}\in\mathcal{W}\}$, then by Definition 1,

[TABLE]

∎

Appendix D Proof of Theorem 2

In this section, we provide the proof of Theorem 2 for model misspecification. First, we slightly modify Lemma 4 and reprove it in model misspecification case.

Lemma 9** (Single step optimization error for misspecification).**

Assume that our function class $\mathcal{F}$ satisfies Assumption 3. Consider a fixed epoch $m\in[M]\backslash[M_{0}]$. We define

[TABLE]

as in Algorithm 1. Also, for any function $V:\mathcal{S}\to[0,H]$, we define

[TABLE]

and

[TABLE]

Then, for any function $V:\mathcal{S}\to[0,H]$ and $\delta\in(0,1)$, there exists an event $\mathcal{E}_{V,\delta}$ where $\Pr\{\mathcal{E}_{V,\delta}\}\geq 1-\delta$, s.t. conditioned on $\mathcal{E}_{V,\delta}$, for any $V^{\prime}:\mathcal{S}\to[0,H]$ with $\|V-V^{\prime}\|_{\infty}\leq 1/T$, we have

[TABLE]

for some constant $c^{\prime}>0$.

Proof.

For any $V:\mathcal{S}\to[0,H]$, we define

[TABLE]

and now we consider a fixed $V$. Note that under Assumption 3, it does not necessary hold that $f_{V}\in\mathcal{F}$, but it can be ensured that

[TABLE]

For any $f\in\mathcal{F}$, define

[TABLE]

By the same method as in Lemma 4, we can prove that with probability at least $1-\delta$,

[TABLE]

Let $\mathcal{E}_{V,\delta}$ denote the above event, and for the rest of the proof, we condition on $\mathcal{E}_{V,\delta}$.

Similarly, by the same method as in Lemma 4, for any $f\in\mathcal{F}$, we have

[TABLE]

For any $V^{\prime}:\mathcal{S}\to[0,H]$ with $\|V^{\prime}-V\|_{\infty}\leq 1/T$, we can obtain that

[TABLE]

Furthermore, again by the same method as in Lemma 4, we can obtain that for any $f\in\mathcal{F}$,

[TABLE]

If we let $f=\hat{f}_{V^{\prime}}=\arg\min_{f\in\mathcal{F}}\|f\|_{\mathcal{D}_{V^{\prime}}^{m}}$, we have

[TABLE]

Now let $\tilde{f}_{V^{\prime}}={\arg\min}_{f\in\mathcal{F}}\|f-f_{V^{\prime}}\|_{\mathcal{Z}^{m}}^{2}$, then

[TABLE]

Therefore,

[TABLE]

which implies

[TABLE]

for some constant $c^{\prime}>0$. ∎

Using the above lemma, we can obtain the following lemma similar to Lemma 5.

Lemma 10** (Confidence region for misspecification).**

Assume that our function class $\mathcal{F}$ satisfies Assumption 3. In Algorithm 1, for $m>M_{0}$, define confidence region

[TABLE]

Then with probability at least $1-\delta/16$, for all $(h,m)\in[H]\times([M]\backslash[M_{0}])$,

[TABLE]

given

[TABLE]

for some constant $c^{\prime}>0$. Here, $\mathcal{W}$ is given in Proposition A.2.

Proof.

The proof is almost identical to that of Lemma 5. ∎

Proof of Theorem 2.

By Lemma 10, Lemma 6, Lemma 7, Lemma 8, the proof is almost the same as the proof of Theorem 1. ∎