Optimistic Planning by Regularized Dynamic Programming

TL;DR

This paper introduces a regularized dynamic programming approach for optimistic planning in infinite-horizon discounted MDPs, enabling efficient learning of near-optimal policies with theoretical guarantees.

Contribution

It presents a novel regularization technique that simplifies analysis and extends optimistic planning to linear function approximation in MDPs.

Findings

Achieves near-optimal statistical guarantees in linear mixture MDPs.

Provides a computationally efficient algorithm for policy learning.

Recovers known guarantees in tabular MDPs.

Abstract

We propose a new method for optimistic planning in infinite-horizon discounted Markov decision processes based on the idea of adding regularization to the updates of an otherwise standard approximate value iteration procedure. This technique allows us to avoid contraction and monotonicity arguments typically required by existing analyses of approximate dynamic programming methods, and in particular to use approximate transition functions estimated via least-squares procedures in MDPs with linear function approximation. We use our method to recover known guarantees in tabular MDPs and to provide a computationally efficient algorithm for learning near-optimal policies in discounted linear mixture MDPs from a single stream of experience, and show it achieves near-optimal statistical guarantees.

Equations

$$V^{\pi }(x)$$

$$Q^{\pi }(x,a)$$

$$V^{\pi }\left(x\right)$$

$$Q^{\pi }$$

$${\nu }^{\pi }\left(x\right)=\left(1-\gamma \right)\sum _{t=0}^{\infty }{\gamma }^t{\mathbb{P}}_{\pi }\left[X_t=x\right].$$

$$E^{\mathsf{T}}{\mu }^{\pi }=\gamma P^{\mathsf{T}}{\mu }^{\pi }+\left(1-\gamma \right){\nu }_0.$$

$$\mathfrak{R}_{T}=\mathbb{E}\left[\sum_{t=1}^{T}\bigl{(}\left\langle\mu^{*}-\mu^{\pi_{t}},r\right\rangle\bigr{)}\right].$$

$$\mathbb{E}\bigl{[}\left\langle\mu^{*}-\mu^{\pi_{\text{out}}},r\right\rangle\bigr{]}=\frac{\mathfrak{R}_{T}}{T},$$

$$Q_{k+1}(x,a)$$

$$V_{k+1}(x)$$

$${\pi }_{k+1}(a|x)$$

$$\bigl{|}\bigl{\langle}\gamma P\left(\cdot\middle|x,a\right)-\gamma\widehat{P}_{k}\left(\cdot\middle|x,a\right),V_{k}\bigr{\rangle}\bigr{|}\leq\textup{CB}_{k}\left(x,a\right).$$

$${\mathfrak{R}}_T$$

$$r+\gamma P{V}_k\le Q_{k+1}\le r+2{\text{CB}}_k+\gamma P{V}_k,$$

$$Q_{k+1}\left(x,a\right)=H=1+\gamma H\geq r\left(x,a\right)+\gamma\bigl{(}PV_{k}\bigr{)}\left(x,a\right).$$

$$Q_{k+1}\left(x,a\right)$$

$$\ge r\left(x,a\right)+\gamma \left(P{V}_k\right)\left(x,a\right),$$

$$Q_{k+1}\left(x,a\right)=0\le r\left(x,a\right)+\gamma \left(P{V}_k\right)\left(x,a\right),$$

$$Q_{k+1}\left(x,a\right)$$

$$\displaystyle\leq r\left(x,a\right)+2\textup{CB}_{k}\left(x,a\right)+\gamma\bigl{(}PV_{k}\bigr{)}\left(x,a\right),$$

$$\sum _{k=1}^K\left(⟨{\mu }^{\ast },r⟩-⟨{\mu }^{{\pi }_k},r⟩\right)\le$$

$$+\frac{1}{\eta }\mathcal{H}\left({\pi }^{\ast }\Vert {\pi }_0\right)+\frac{\eta H^{3}K}{2}.$$

$$\sum _{k=1}^K\left(⟨{\mu }^{\ast },r⟩-⟨{\mu }^{{\pi }_k},r⟩\right)=\sum _{k=1}^K\left({\Delta }_k^{\ast }+{\Delta }_k\right),$$

$$\begin{array}{rl}{\Delta }_k^{\ast }& =⟨{\mu }^{\ast },r⟩-\left(1-\gamma \right)⟨{\nu }_0,V_k⟩\\ & =⟨{\mu }^{\ast },r+\gamma P{V}_k⟩-⟨{\mu }^{\ast },E{V}_k⟩.\end{array}$$

$${\Delta }_k^{\ast }$$

$$=⟨{\mu }^{\ast },Q_{k+1}-E{V}_{k+1}⟩+⟨{\mu }^{\ast },E{V}_{k+1}-E{V}_k⟩.$$

$$\sum _{k=1}^K{\Delta }_k^{\ast }$$

$$+⟨{\mu }^{\ast },E\left(V_{K+1}-V_1\right)⟩,$$

$${\overline{V}}_k\left(x\right)=\underset{p\in \Delta \left(\mathcal{A}\right)}{\max }\left\{⟨p,{\overline{Q}}_k\left(x,\cdot \right)⟩-\frac{1}{\eta }{\mathcal{D}}_{\text{KL}}\left(p\Vert {\pi }_0\left(\cdot |x\right)\right)\right\}$$

$$\displaystyle\qquad\geq\left\langle\pi^{*}\left(\cdot\middle|x\right),\overline{Q}_{k}\left(x,\cdot\right)\right\rangle-\frac{1}{\eta}\mathcal{D}_{\text{KL}}\bigl{(}\pi^{*}(\cdot|x)\|\pi_{0}(\cdot|x)\bigr{)}.$$

$$\sum _{k=1}^K{\Delta }_k^{\ast }$$

$${\Delta }_k$$

$$\displaystyle\leq\left(1-\gamma\right)\left\langle\nu_{0},V_{k}\right\rangle-\bigl{\langle}\mu^{\pi_{k}},Q_{k+1}-2\textup{CB}_{k}-\gamma PV_{k}\bigr{\rangle}$$

$$=⟨E^{\mathsf{T}}{\mu }^{{\pi }_k}-\gamma P^{\mathsf{T}}{\mu }^{{\pi }_k},V_k⟩$$

$$-⟨{\mu }^{{\pi }_k},Q_{k+1}-\gamma P{V}_k⟩+2⟨{\mu }^{{\pi }_k},{\text{CB}}_k⟩,$$

$${\Delta }_k$$

$$=⟨{\mu }^{{\pi }_k},E{V}_k⟩-⟨{\mu }^{{\pi }_{k+1}},Q_{k+1}⟩$$

Full text

Machine Learning, ICML

1 Introduction

The idea of constructing a confidence set of statistically plausible models and picking a policy that maximizes the expected return in the best of these models can be traced back to the pioneering work of Lai & Robbins (1985) in the context of multi-armed bandit problems, and has been successfully extended to address the exploration-exploitation dilemma in reinforcement learning (RL, Sutton & Barto, 2018). This popular design principle came to be known as optimism in the face of uncertainty, and the associated optimization task as the problem of optimistic planning. The optimistic principle has driven the development of statistically efficient RL algorithms for a variety of problem settings. Following the work of Brafman & Tennenholtz (2002); Strehl et al. (2009) on optimistic exploration methods for RL in Markov decision processes (MDPs), a breakthrough was achieved by Jaksch, Ortner, and Auer (2010), whose UCRL2 algorithm was shown to achieve near-optimal regret guarantees in a broad class of tabular MDPs. In subsequent years, their work inspired an impressive amount of follow-up work, leading to a variety of extensions, improvements, and other mutations.

The computational efficiency of such optimistic methods crucially hinges on the implementation of the optimistic planning subroutine. In the work of Jaksch et al. (2010), this was addressed by a procedure called extended value iteration (EVI), which performs dynamic programming (DP) in an auxiliary MDP where the confidence set of models is projected to the space of actions, allowing the realization of arbitrary transitions that are statistically plausible given all past experience. After mild adjustments, the EVI procedure can be shown to give near-optimal solutions to the optimistic planning problem in a computationally efficient manner (cf. Fruit et al., 2018 and Section 38.5.2 in Lattimore & Szepesvári, 2020). Other, even more effective optimistic dynamic programming procedures have been proposed and analyzed (Fruit et al., 2018; Qian et al., 2018). However, these computational developments have been largely restricted to the relatively simple tabular setting.

In recent years, the RL theory literature has seen a massive revival largely due to the breakthrough achieved by Jin, Yang, Wang, and Jordan (2020), who successfully extended the idea of optimistic exploration to a class of large-scale MDPs using linear function approximation. While extremely influential, their approach (and virtually all of its numerous follow-ups) are limited to the relatively simple setting of finite-horizon MDPs. The reason for this limitation is inherent in their algorithm design that crucially uses the fact that optimistic planning in finite-horizon MDPs can be solved via a simple backward recursion over the time indices within each episode (Neu & Pike-Burke, 2020). This idea completely fails for infinite-horizon problems where dynamic programming methods should aim to approximate the solution of a fixed-point equation. Solving such fixed-point equations is possible in the tabular case but no known efficient method exists for linear function approximation, the short reason being that the least-squares transition estimator used in the construction of Jin et al. (2020) cannot be straightforwardly used to build an approximate Bellman operator that satisfies the necessary contraction properties.

The best attempt at tackling the infinite-horizon setting under function approximation we are aware of is by Wei, Jahromi, Luo, and Jain (2021), who propose algorithms that are either statistically or computationally efficient, but fall short of providing an algorithm with both of these desired properties. Another good contribution was made by Vial, Parulekar, Shakkottai, and Srikant (2022), who provided approximate DP methods for stochastic shortest path problems with linear transition functions, and analyzed them via studying the concentration properties of the empirical transition operator. This technique allowed them to prove regret bounds, but the guarantees did not reach optimality in terms of scaling with the time horizon unless strong assumptions are made. Notably, Vial et al. (2022) only managed to perform a tight analysis in the special case where the features are orthogonal, which allowed them to reason about contraction properties of the empirical Bellman operator. Lacking a general contraction argument, or another idea that would enable computationally efficient optimistic planning, efficient exploration-exploitation in infinite-horizon MDPs under function approximation has remained unsolved so far.

This is the problem we address in this paper in the context of discounted infinite-horizon MDPs. Instead of relying on a contraction argument (or an approximate version thereof), we propose to solve the optimistic planning problem using regularized dynamic programming. In particular, we consider a variant of the Mirror-Descent Modified Policy Iteration (MD-MPI) algorithm of Geist, Scherrer, and Pietquin (2019) that uses a least-squares estimator of the transition kernel and an exploration bonus to define an optimistic regularized Bellman operator. Using arguments from the classic analysis of mirror descent methods, we show that each application of this optimistic operator improves the quality of the policy up to an additive error term that telescopes over the iterations. In other words, we show that each iteration improves over the last one in an average sense. This is in stark contrast to arguments used for analyzing previous optimistic planning methods that relied on contraction arguments which guarantee strict improvements to the policy in each iteration. The advantage is that it remains applicable even when the approximate dynamic programming operator is not contractive or monotone (even approximately).

Our concrete contribution is applying the above scheme to discounted linear mixture MDPs and showing that it achieves a near-optimal regret bound of order $\sqrt{\left(B^{2}dH+d^{2}H^{3}+\log\left|\mathcal{A}\right|H^{4}\right)T}$, where $d$ is the feature dimension, $B$ is a bound on the norm of the features, and $H=\frac{1}{1-\gamma}$ is the effective horizon. This result implies that our algorithm produces an $\varepsilon$-optimal policy after about $\left(B^{2}dH+d^{2}H^{3}+\log\left|\mathcal{A}\right|H^{4}\right)/\varepsilon^{2}$ iterations. Each policy update takes poly($d,H,T$) iterations of regularized dynamic programming, each consisting of poly($d,H,T$) elementary operations. This is to be contrasted with previous contributions on a similar111We provide a detailed discussion about the differences between our settings in Section 6.1. setting by Zhou, He, and Gu (2021), whose policy updates rely on a version of EVI adapted to linear function approximation. Their EVI variants require globally constraining the model parameters in a way that the model is a valid transition kernel. While this last constraint allowed them to reason about contractive properties of the EVI iterations, it is practically impossible to enforce without making strong assumptions on the feature maps and the MDP itself. The difficulty remains even when the property is only required to hold locally in each state. In this sense, our method is the first to obtain near-optimal statistical rates while also being entirely computationally feasible.

The rest of this paper is organized as follows. After presenting the notation at the end of this section and the background in Section 2, we introduce our algorithmic framework in Section 3. We provide generic performance guarantees and explain the key steps of the analysis in Section 4. The guarantees are instantiated in the context of tabular and linear mixture MDPs in Section 5. We conclude in Section 6 with a discussion of our contribution along with its limitations.

Notation.

For a natural number $N>0$, we denote $\left[N\right]=\left\{1,2,\dots,N\right\}$. For a real number $M$, we define the truncation operator $\Pi_{M}$ that acts on functions $f$ defined on a domain $A$ via $\Pi_{M}f:x\mapsto\max\left(\min\left[f\left(x\right),M\right],0\right)$. For a measurable space $(A,\mathcal{F})$, we define the set of all probability distributions $\Delta(A)$, and for any two distributions $P,Q\in\Delta(A)$ such that $P\ll Q$, we define the relative entropy as $\mathcal{D}_{\text{KL}}\left(P\|Q\right)=\mathbb{E}_{a\sim P}\left[\ln\left(\frac{\mathrm{d}P}{\mathrm{d}Q}(a)\right)\right]$. For a distribution $P\in\Delta(A)$ and a bounded function $f\in^{A}$, we write $\left\langle P,f\right\rangle=\mathbb{E}_{a\sim P}\left[f(a)\right]$ to denote the expectation of $f$ under $P$, and we will use the same notation for finite-dimensional vector spaces to denote inner products. For a finite-dimensional vector $v\in^{d}$ and a square matrix $Z\in^{d\times d}$, we will use the notation $\left\|v\right\|_{Z}=\sqrt{\left\langle v,Zv\right\rangle}$.

2 Preliminaries

We consider a discounted MDP $\mathcal{M}=\left(\mathcal{X},\mathcal{A},r,P,\gamma,\nu_{0}\right)$, where $\mathcal{X}$ is the finite state space222Our results extend to the case where $\mathcal{X}$ is a measurable space. The precise definitions require measure-theoretic concepts (Bertsekas & Shreve, 1996). For the sake of readability and because they are well understood, we only consider finite state spaces here., $\mathcal{A}$ is the finite action space, $r:\mathcal{X}\times\mathcal{A}\rightarrow\left[0,1\right]$ is the deterministic reward function assumed to be known333It is a standard assumption, and removing it only costs a constant factor in the regret (Jaksch et al., 2010)., $P:\mathcal{X}\times\mathcal{A}\rightarrow\Delta\left(\mathcal{X}\right)$ is the transition probability distribution, $\gamma\in\left(0,1\right)$ is the discount factor, and $\nu_{0}\in\Delta\left(\mathcal{X}\right)$ is the initial state distribution. The model describes a sequential interaction scheme between a decision-making agent and its environment, where the following steps are repeated for a sequence of rounds $t=1,2,\dots$ after the initial state is drawn as $X_{0}\sim\nu_{0}$: the agent observes the state $X_{t}\in\mathcal{X}$, selects an action $A_{t}\in\mathcal{A}$, obtains reward $r\left(X_{t},A_{t}\right)$, and the environment generates the next state $X_{t+1}\sim P\left(\cdot\middle|X_{t},A_{t}\right)$. The goal of the agent is to pick its sequence of actions in a way that the total discounted return $\sum_{t=0}^{\infty}\gamma^{t}r\left(X_{t},A_{t}\right)$ is as large as possible.

Below we describe the most fundamental objects relevant to our work, and refer the reader to the classic book of Puterman (2014) for more context and details. A (stationary) policy is a mapping $\pi:\mathcal{X}\rightarrow\Delta\left(\mathcal{A}\right)$ from a state to a probability measure over actions. The value function and action-value function of a policy $\pi$ are respectively defined as the functions $V^{\pi}\in^{\mathcal{X}}$ and $Q^{\pi}\in^{\mathcal{X}\times\mathcal{A}}$ mapping each state $x$ and state-action pair $x,a$ to

[TABLE]

where $\mathbb{E}_{\pi}$ denotes the expectation with respect to the probability measure $\mathbb{P}_{\pi}$, generated by the interaction between the environment and the policy $\pi$. With some abuse of notation, we define the conditional expectation operator $P:^{\mathcal{X}}\rightarrow^{\mathcal{X}\times\mathcal{A}}$ as $\left(Pf\right)\left(x,a\right)=\sum_{x^{\prime}\in\mathcal{X}}P\left(x^{\prime}\middle|x,a\right)f\left(x^{\prime}\right)$, for $f\in^{\mathcal{X}}$. Its adjoint $P^{\mathsf{\scriptscriptstyle T}}$ acts on distributions $\mu\in\Delta(\mathcal{X}\times\mathcal{A})$ as $\left(P^{\mathsf{\scriptscriptstyle T}}\mu\right)\left(x^{\prime}\right)=\sum_{x,a\in\mathcal{X}\times\mathcal{A}}P\left(x^{\prime}|x,a\right)\mu\left(x,a\right)$. It returns the state distribution realized after starting from the state-action distribution $\mu$ and then taking a step forward in the MDP dynamics. With these, we can simply state the Bellman equations tying together the value functions as

[TABLE]

We also introduce the operator $E:^{\mathcal{X}}\rightarrow^{\mathcal{X}\times\mathcal{A}}$ acting on functions $f\in^{\mathcal{X}}$ via the assignment $\left(Ef\right)\left(x,a\right)=f\left(x\right)$, and its adjoint via its action $E^{\mathsf{\scriptscriptstyle T}}\mu\left(x\right)=\sum_{a}\mu\left(x,a\right)$ on distributions $\mu\in\Delta(\mathcal{X}\times\mathcal{A})$. The operator $E$ can be thought of as a “padding” operator over the action space that allows us to use vector notation, while $E^{\mathsf{\scriptscriptstyle T}}$ applied to a state-action distribution returns the corresponding marginal distribution of states. The adjoint $P^{\mathsf{\scriptscriptstyle T}}$ (resp. $E^{\mathsf{\scriptscriptstyle T}}$) is the operator such that, for any $f,g$, $\left\langle Pf,g\right\rangle=\left\langle f,P^{\mathsf{\scriptscriptstyle T}}g\right\rangle$ (resp. $E$, $E^{\mathsf{\scriptscriptstyle T}}$).

In a discounted MDP, a policy $\pi$ induces a unique normalized discounted occupancy measure over the state space, defined for any state $x\in\mathcal{X}$ as

[TABLE]

The normalization term $\left(1-\gamma\right)$ guarantees $\nu^{\pi}$ is a probability measure over $\mathcal{X}$. We call the inverse of this normalization constant the effective horizon and denote it by $H=\frac{1}{1-\gamma}$. We also define the associated state-action occupancy measure $\mu^{\pi}$, defined as $\mu^{\pi}\left(x,a\right)=\nu^{\pi}\left(x\right)\pi\left(a\middle|x\right)$. State-action occupancy measures are known to satisfy the following recurrence relation that is sometimes called the system of Bellman flow constraints:

[TABLE]

Using the state-action occupancy measure, the discounted return of a policy can be written as $R_{\gamma}^{\pi}=\frac{1}{1-\gamma}\left\langle\mu^{\pi},r\right\rangle$. We will use $\mu^{*}$ to denote an occupancy measure with maximal return and $\nu^{*}=E^{\mathsf{\scriptscriptstyle T}}\mu^{*}$ to denote the associated state-occupancy measure. Finally, given two policies $\pi,\pi^{\prime}$, we denote $\mathcal{D}_{\text{KL}}\left(\pi\|\pi^{\prime}\right)=\left(\mathcal{D}_{\text{KL}}\left(\pi\left(\cdot\middle|x\right)\|\pi^{\prime}\left(\cdot\middle|x\right)\right)\right)_{x\in\mathcal{X}}$, and we define $\mathcal{H}\left(\pi\|\pi^{\prime}\right)=\left\langle\nu^{\pi},\mathcal{D}_{\text{KL}}\left(\pi\|\pi^{\prime}\right)\right\rangle$, the conditional relative entropy444Technically, this is the conditional relative entropy between the occupancy measures $\mu^{\pi}$ and $\mu^{\pi^{\prime}}$, but we will keep referring to it in terms of the policies to keep our notations light. We refer to Neu et al. (2017) for further discussion..

In this paper, we will consider the setting of online learning in discounted MDPs, where the agent aims to produce an $\varepsilon$-optimal policy $\pi_{\text{out}}$ satisfying $\left\langle\mu^{*}-\mu^{\pi_{\text{out}}},r\right\rangle\leq\varepsilon$ based on a single stream of experience in the MDP. We will assume that the learner has access to a reset action that drops the agent back to a state randomly drawn from the initial-state distribution $\nu_{0}$, and that the learner follows a stationary policy $\pi_{t}$ in each round $t$. We will measure the performance in terms of the number of samples necessary to guarantee that the output policy is $\varepsilon$-optimal. As an auxiliary performance measure, we will also consider the expected regret (or simply, regret)555In the related literature, it is more common to define regret as a random variable and bound it with high probability. Our algorithm is only suitable for bounding the expected regret, and thus we only define this quantity here; we defer further discussion to Section 6. of the learner defined as

[TABLE]

It is easy to see that a regret bound can be converted into sample complexity guarantees. In particular, selecting a time index $I$ uniformly at random from $1,\dots,T$ and returning $\pi_{\text{out}}=\pi_{I}$ guarantees that

[TABLE]

which can be made arbitrarily small if $\mathfrak{R}_{T}$ grows sublinearly and $T$ is set large enough. We note here that, while superficially similar to the discounted regret criterion considered in earlier works like Liu & Su (2020); He et al. (2021) or Zhou et al. (2021), there are some major differences between our objectives. We only point out here that we consider the complexity of producing a good policy to execute from the initial state distribution, whereas theirs measures the suboptimality of the policies along the trajectory traversed by the learner. We defer a further discussion of the two settings to Section 6.1.

3 Algorithm

Our approach implements the principle of optimism in the face of uncertainty in discounted MDPs. Instead of aiming to solve an optimistic version of the Bellman optimality equations via extended value iteration as done by Jaksch et al. (2010), our method draws on techniques from convex optimization aiming at average policy improvement. In particular, our planning procedure is based on a regularized version of approximate value iteration and incorporates an optimistic estimate of the associated Bellman operator. Consequently, we refer to our algorithm as RAVI-UCB, standing for Regularized Approximate Value Iteration with Upper Confidence Bounds.

RAVI-UCB performs a sequence of regularized Q-function and policy updates as follows. Starting with an initial estimate $V_{0}=0$ and an initial policy $\pi_{0}$, it calculates a sequence of updates for $k=1,\dots,K$ as

[TABLE]

Here, $\widehat{P}$ is a nominal transition model and $\textup{CB}_{k}$ is an exploration bonus defined to be large enough to ensure that $\gamma\widehat{P}V_{k}+\textup{CB}_{k}\geq\gamma PV_{k}$ and so that $Q_{k+1}$ is an upper bound on the regularized Bellman update $r+\gamma PV_{k}$. The Q-functions are truncated to the range $\left[0,H\right]$ to make sure that the optimistic property above can be ensured by setting a reasonably sized exploration bonus $\textup{CB}_{k}$. It is important to note that $Q_{k+1}$ does not directly attempt to approximate the optimal action-value function $Q^{*}$ in the true MDP, which marks a clear departure from previously known optimism-based regret analyses. Instead, our analysis will show that $\left(1-\gamma\right)\left\langle\nu_{0},V_{k}\right\rangle$ acts as an optimistic estimate of the optimal return $\left(1-\gamma\right)\left\langle\nu_{0},V^{*}\right\rangle$ in an “average” sense, and that the total reward of our algorithm can also be bounded in terms of the same quantity.

The overall procedure is presented as Algorithm 1. The algorithm proceeds in a sequence of epochs $k=1,2,\dots$, where a new epoch is started by taking the reset action with probability $1-\gamma$ in each round, which results in epochs of average length $H=\frac{1}{1-\gamma}$. At the beginning of each epoch, we update the model estimate $\widehat{P}_{k}$ and perform one step of online mirror descent to produce the new policy $\pi_{k}$ and the associated softmax value function $V_{k}$. We then update the exploration bonuses $\textup{CB}_{k}$ such that they satisfy, for all $x,a$

[TABLE]

We will refer to exploration bonuses satisfying the above condition as valid. As we will see explicitly in Section 5, the model estimate and bonuses are computed using the data gathered so far, $\mathcal{D}_{T_{k}}$, where $T_{k}$ denotes the first time index of epoch $k$. Finally, we apply an optimistic Bellman update to produce a state-action value estimate $Q_{k+1}$.

We highlight that the assignments in Algorithm 1 are only made symbolically for all $x,a$, and a practical implementation will not necessarily need to loop over the entire state-action space. Rather, all quantities of interest can be computed on demand while executing the policy in runtime.

Finally, to make some of the arguments in Section 4 more convenient to state, we introduce some notation. We let $\mathcal{T}_{k}=\left\{T_{k},T_{k}+1,\dots,T_{k+1}-1\right\}$ denote the time indices belonging to epoch $k$, and $K\left(T\right)$ denote the total number of epochs. For the sake of analysis, we pad out the trajectory of states and actions with the artificial observations $\left(X_{T+1},A_{T+1},\dots,X_{T^{+}},A_{T^{+}}\right)$, where $T^{+}$ is the first time that a reset would have occurred had the algorithm been executed beyond time step $T$. These observations are well-defined random variables, and are introduced to make sure that the last epoch does not require special treatment.

4 Main Result & Analysis

Our main technical result regarding the performance of RAVI-UCB is the following regret bound.

Theorem 4.1**.**

Let $\left\{\pi_{k}\right\}_{k}$ and $\left\{\textup{CB}_{k}\right\}_{k}$ be the policies and exploration bonuses produced by RAVI-UCB over $T$ timesteps, where the input is $\eta=\sqrt{2\log\left|\mathcal{A}\right|/\left(H^{2}T\right)}$, $V_{0}=0$ and any policy $\pi_{0}$. Suppose that the sequence of bonuses $\left\{\textup{CB}_{k}\right\}_{k}$ is valid in the sense of Equation (2). Then the policies $\left\{\pi_{k}\right\}_{k}$ satisfy the following bound:

[TABLE]

We present the proof of Theorem 4.1 below. In particular, we state a sequence of lemmas whose combination will yield the complete proof. We will provide the proofs that we believe to be most insightful in the main text, and relegate the more technical ones to Appendix A.

The analysis will be split into two main parts: one pertaining to the general properties of our optimistic planning procedure and to the eventual regret bound that can be derived from it, and one concerning the specifics of the setting considered. In particular, we first analyze RAVI-UCB using a generic exploration bonus that we will suppose to be “valid”, and then show in Section 5 how to derive such valid exploration bonuses in the concrete settings of tabular MDPs and linear mixture MDPs.

4.1 Optimistic Planning

We first study the properties of our optimistic planning procedure, without making explicit references to the setting. For this general analysis, we will fix an epoch index $k$, assume that $\widehat{P}_{k}$ is some estimator of the transition kernel $P$ and that the exploration bonus $\textup{CB}_{k}$ is valid in the sense of Equation (2). We provide the following inequality that will be useful for bounding the suboptimality gaps.

Lemma 4.2**.**

Let $Q_{k+1}$ be the state-action value estimate produced by RAVI-UCB in epoch $k$, with any input, and assume the bonuses $\textup{CB}_{k}$ are valid in the sense of Equation (2). Then,

[TABLE]

where $V_{k}$ is the value estimate defined in Algorithm 1.

Proof.

We start by proving the lower-bound. For each state-action pair $\left(x,a\right)$, we need to handle two separate cases corresponding to whether or not $Q_{k+1}\left(x,a\right)$ is truncated from above. In the first case, we have $Q_{k+1}\left(x,a\right)=H$, which implies

[TABLE]

Here, we have crucially used the condition $V_{k}\leq H$ in the inequality, which was made possible by truncating the Q-functions to the range $\left[0,H\right]$. In the other case where $Q_{k+1}\left(x,a\right)\leq H$, we use the validity of $\textup{CB}_{k}$ to show the following inequality:

[TABLE]

where the first inequality is valid even when a truncation from below happens.

For the upper-bound, we proceed similarly and consider the two cases corresponding to whether or not $Q_{k+1}\left(x,a\right)$ is truncated from below in each state-action pair. First considering the case where $Q_{k+1}\left(x,a\right)=0$, we observe that

[TABLE]

from which the claim follows due to non-negativity of $\textup{CB}_{k}$. As for the other case, we have

[TABLE]

where the last step follows from the validity condition on $\textup{CB}_{k}$. ∎

Our key result regarding the quality of the policies produced by RAVI-UCB is the following.

Lemma 4.3**.**

Let $K$ be a fixed number of epochs, and let $\pi_{k}$ and $\textup{CB}_{k}$ be the policy and exploration bonus produced by RAVI-UCB in epoch $k$, where the input is $V_{0}=0$, any policy $\pi_{0}$, and any $\eta>0$. Suppose that $\left\{\textup{CB}_{k}\right\}_{k}$ is a sequence of valid exploration bonuses in the sense of Equation (2). Then, the sequence $\left\{\pi_{k}\right\}_{k}$ satisfies the following bound:

[TABLE]

Proof.

The main idea of the proof is to show that, under the validity condition of the exploration bonuses, $\left(1-\gamma\right)\left\langle\nu_{0},V_{k}\right\rangle$ acts as an approximate upper bound on the optimal return $\left\langle\mu^{*},r\right\rangle$, up to some additional terms resulting from the use of incremental updates. Thanks to the use of regularization, we can show that these additional terms are small on average, and that the gap between the optimistic value and the return of $\pi_{k}$ can be bounded in terms of $\left\langle\mu^{\pi_{k}},\textup{CB}_{k}\right\rangle$. With this in mind, we begin by rewriting the performance gap of the output policy as follows:

[TABLE]

where we defined $\Delta_{k}^{*}=\left\langle\mu^{*},r\right\rangle-\left(1-\gamma\right)\left\langle\nu_{0},V_{k}\right\rangle$ and $\Delta_{k}=\left(1-\gamma\right)\left\langle\nu_{0},V_{k}\right\rangle-\left\langle\mu^{\pi_{k}},r\right\rangle$ for all $k$.

Let us now fix some $k$ and consider the first term, $\Delta_{k}^{*}$. We start by observing that $\left(1-\gamma\right)\nu_{0}=E^{\mathsf{\scriptscriptstyle T}}\mu^{*}-\gamma P^{\mathsf{\scriptscriptstyle T}}\mu^{*}$, which allows us to write

[TABLE]

In order to treat the first term in Equation (4), we use the lower-bound from Lemma 4.2 to obtain

[TABLE]

Summing up for all $k=1,\dots,K$, we get

[TABLE]

where we defined $\overline{Q}_{k}=\sum_{i=1}^{k}Q_{i}$ and $\overline{V}_{k}=\sum_{i=1}^{k}V_{i}$ for any $k$. By a classic telescoping argument (presented in Lemma C.1), one can show that, for all $k$,

[TABLE]

Combining this with the previous inequality, we get

[TABLE]

by definition of the conditional entropy and $V_{1}=0$.

We now move on to bounding $\Delta_{k}$. Then, using the upper-bound of Lemma 4.2 to lower-bound $r$, we bound $\Delta_{k}$ as follows:

[TABLE]

where we have used $\left(1-\gamma\right)\nu_{0}=E^{\mathsf{\scriptscriptstyle T}}\mu^{\pi_{k}}-\gamma P^{\mathsf{\scriptscriptstyle T}}\mu^{\pi_{k}}$ in the third line. We can then rewrite the current upper-bound as

[TABLE]

To proceed, we use Lemma C.1 to note that

[TABLE]

which allows us to continue as

[TABLE]

The last remaining difficulty is to control the second difference in the last inequality. This can be done thanks to the regularization, that makes the occupancy measures change “slowly enough”. To proceed, we use Pinsker’s inequality and the boundedness of $Q_{k+1}$ to show

[TABLE]

Appealing to Lemma A.1, we can bound the last term as

[TABLE]

Using these results, we obtain

[TABLE]

where the last step follows from the Fenchel–Young inequality applied to the convex function $f\left(z\right)=z^{2}/2$. Then, summing up both sides of Equation (6) for all $k=1,\dots,K$,

[TABLE]

where we used $V_{1}=0$. Combining Equations (5) and (7),

[TABLE]

where we used $\left\langle\mu^{*}-\mu^{\pi_{K+1}},EV_{K+1}\right\rangle\leq 2H$. ∎

4.2 The Epoch Schedule

The final part is to account for the effects of the randomized epoch schedule. Provided that the exploration bonuses are valid, we need to control the sum $\sum_{t=1}^{T}\left\langle\mu^{\pi_{t}},\textup{CB}_{t}\right\rangle$. We relate it to a more easily tractable sum in the next lemma.

Lemma 4.4**.**

The sequence of policies selected by RAVI-UCB satisfies

[TABLE]

The proof is in Appendix A.3. This bound is guaranteed by the epoch schedule used by RAVI-UCB that ensures that within each epoch $k$ of geometric length, the sequence of realized state-action trajectory is distributed according to the occupancy measure of $\pi_{k}$.

4.3 Putting Everything Together

The proof of Theorem 4.1 concludes by combining the above claims. In anticipation of Section 5, for our main assumption to be satisfied we let $\delta=1/T$ and define the exploration bonuses as in Lemma 5.1 or Lemma 5.4. This implies the resulting exploration bonuses are valid with probability at least $1-\delta$, so on this event we can use Lemma 4.3 to bound the expected regret of RAVI-UCB. Setting $\pi_{0}$ as the uniform policy, we get

[TABLE]

where we used that the expected epoch length is $H$ and $\mathcal{H}\left(\pi^{*}\|\pi_{0}\right)\leq\log\left|\mathcal{A}\right|$. Noticing that $\mathbb{E}\left[K\right]=\left(1-\gamma\right)T$ and setting the learning rate $\eta=\sqrt{2\log\left|\mathcal{A}\right|/\left(H^{2}T\right)}$, the expected optimization error becomes

[TABLE]

The remaining terms in the regret bound corresponding to the sum of exploration bonuses can be bounded by appealing to Lemma 4.4. This concludes the proof.

5 Applications

We now consider two classes of MDPs and show how to implement our algorithm and derive a regret bound.

5.1 Tabular MDPs

For didactic purposes, we first instantiate RAVI-UCB in the setting of tabular MDPs with small state and action spaces. As we will see, a simple application of our framework allows us to recover known guarantees in this setting. The algorithm can be found in Appendix B.1. Let $N_{1}(x,a)=1$ and $N_{1}^{\prime}(x,a,x^{\prime})=0$ denote the initial counts666We initialize $N_{1}$ at 1 to avoid divisions by zero. for the tuples $\left(x,a\right)$ and $(x,a,x^{\prime})$. At epoch $k$, for $t\in\mathcal{T}_{k}$, we update $\mathcal{D}_{t+1}=\left(N_{t+1},N_{t+1}^{\prime}\right)$ as $N_{t+1}(x,a)=N_{t}(x,a)+\mathbb{I}_{\left\{X_{t}=x,A_{t}=a\right\}}$ and $N_{t+1}^{\prime}(x,a,x^{\prime})=N_{t}^{\prime}(x,a,x^{\prime})+\mathbb{I}_{\left\{X_{t}=x,A_{t}=a,X_{t+1}=x^{\prime}\right\}}$. We use $\widehat{P}_{k}\left(x^{\prime}\middle|x,a\right)=N_{T_{k}}(x,a,x^{\prime})/N_{T_{k}}(x,a)$ as a model estimate, and given $\beta>0$, the exploration bonuses are defined as

[TABLE]

The following lemma shows that an appropriate choice of the scaling parameter $\beta$ ensures the validity of the exploration bonuses.

Lemma 5.1**.**

Let $\delta\in\left(0,1\right)$. Then, setting the coefficient $\beta=8H\sqrt{\left|\mathcal{X}\right|\log\left(\left|\mathcal{X}\right|\left|\mathcal{A}\right|T/\delta\right)}$ guarantees that, with probability $1-\delta$, the validity condition (2) is satisfied by $\textup{CB}_{k}$ as defined in Equation (8) for all $k$.

Then, we can bound the bonuses as follows.

Lemma 5.2**.**

The sum of exploration bonuses generated by RAVI-UCB satisfies

[TABLE]

We refer the reader to previous works for the proofs of the above two lemmas (see, e.g., Jaksch et al., 2010; Fruit et al., 2018). Combining the above two results gives a regret bound of order ${\left|\mathcal{X}\right|H\sqrt{\left|\mathcal{A}\right|T}}$, as expected.

5.2 Linear Mixture MDPs

We now focus on a class of MDPs commonly referred to as linear mixture MDPs (Modi et al., 2020; Ayoub et al., 2020) formally defined as follows.

Assumption 5.3** (Linear mixture MDP).**

There exist a known feature map $\psi:\mathcal{X}\times\mathcal{A}\times\mathcal{X}\rightarrow\mathbb{R}^{d}$, and an unknown $\theta\in\mathbb{R}^{d}$ with $\left\|\theta\right\|_{2}\leq B$ such that $P(x^{\prime}|x,a)=\sum_{i=1}^{d}\theta_{i}\psi_{i}(x,a,x^{\prime})$. Furthermore, for any $(x,a)\in\mathcal{X}\times\mathcal{A}$, $V\in\left[0,H\right]^{\mathcal{X}}$,

[TABLE]

Here, we suppose $\mathcal{M}$ satisfies Assumption 5.3. While remotely related to the notion of linear MDPs (Jin et al., 2020; Yang & Wang, 2019), linear mixture MDPs are a distinct class of models that cannot be captured in that framework, and have been widely studied in the past few years as linear MDPs—we refer to Zhou et al. (2021) for further discussion. As often assumed in the related literature, we assume the map $\varphi_{k}(x,a)=\sum_{x^{\prime}}\psi(x,a,x^{\prime})V_{k}\left(x^{\prime}\right)$ can be computed (or approximated) efficiently. We provide a detailed discussion of all such computational matters in Section 6.2.

The algorithm is in Appendix B.2. Let $\lambda>0$ be a regularization parameter, $\Lambda_{1}=\lambda I$, and $b_{1}=0$. At epoch $k$, for $t\in\mathcal{T}_{k}$, the data is stored as $\mathcal{D}_{t+1}=\left(\Lambda_{t+1},b_{t+1}\right)$ where $\Lambda_{t+1}=\Lambda_{t}+\varphi_{k}\left(x_{t},a_{t}\right)\varphi_{k}\left(x_{t},a_{t}\right)^{\mathsf{\scriptscriptstyle T}}$ and $b_{t+1}=b_{t}+\varphi_{k}\left(x_{t},a_{t}\right)V_{k}\left(x_{t+1}\right)$. $\widehat{P}_{k}=\sum_{i}\widehat{\theta}_{k,i}\psi_{i}$ is computed via a least-squares regression, where $\widehat{\theta}_{k}=\Lambda_{T_{k}}^{-1}b_{T_{k}}$. Given $\beta>0$, the exploration bonuses are defined as

[TABLE]

We now turn to the validity condition required by Lemma 4.3.

Lemma 5.4**.**

Let $\delta\in\left(0,1\right)$. Then, setting the coefficient $\beta=H\sqrt{2\left(\frac{d}{2}\log\left[1+\frac{TB^{2}H^{2}}{\lambda d}\right]+\log\frac{1}{\delta}\right)}+\sqrt{\lambda}B$ guarantees that, with probability $1-\delta$, the validity condition (2) is satisfied by $\textup{CB}_{k}$ as defined in Equation (9) for all $k$.

The proof is in Appendix A.2. It relies on standard techniques regarding linear mixture MDPs (Zhou et al., 2021; Cai et al., 2020). One important property required is the boundedness of each $V_{k}$ that is guaranteed by the truncation. Then, we can bound the sum of the exploration bonuses with the following lemma.

Lemma 5.5**.**

The sum of exploration bonuses generated by RAVI-UCB satisfies

[TABLE]

The proof (Appendix A.4) follows from a series of small (but somewhat tedious) adjustments of a classic result often referred to as the “elliptical potential lemma”, the main challenge being dealing with the randomized epoch schedule.

Our main technical result regarding the performance of RAVI-UCB is the following.

Theorem 5.6**.**

Suppose that RAVI-UCB is run with the uniform policy as $\pi_{0}$, $V_{0}=0$, $\lambda=1$, a learning rate $\eta=\sqrt{2\log\left|\mathcal{A}\right|/\left(H^{2}T\right)}$, and an exploration parameter $\beta=H\sqrt{2\left(\frac{d}{2}\log\left[1+\frac{TB^{2}H^{2}}{d}\right]+\log T\right)}+B$. Then, the expected regret of RAVI-UCB satisfies

[TABLE]

$\widetilde{\mathcal{O}}\left(\cdot\right)$ hides logarithmic factors of $T$, $B$, $d$, and $H$. A perhaps more useful result is the following, derived from an online-to-batch conversion. Suppose RAVI-UCB returns a policy $\pi_{\text{out}}=\pi_{U}$ with $U$ being an epoch index chosen uniformly at random from the range of epochs. The following corollary provides a guarantee on the quality of this policy.

Corollary 5.7**.**

Let $\varepsilon>0$. Then, RAVI-UCB run with the same parameters as before outputs a policy $\pi_{\text{out}}$ satisfying $\mathbb{E}\left[\left\langle\mu^{*}-\mu^{\pi_{\text{out}}},r\right\rangle\right]\leq\varepsilon$ after $T_{\varepsilon}$ steps, with

[TABLE]

The expectation appearing in the first statement is with respect to the random transitions in the MDP and the epoch scheduling, whereas the expectation in the second one is also with respect to the random choice of the policy. It is possible to remove the former expectation, but the latter is inherent to the online-to-batch conversion process used by our analysis. We will return to this point in Section 6.2.

6 Discussion

We now discuss the merits and limitations of our results, and point out directions for future research.

6.1 Results and Comparisons

There are many differences between our approach and previously proposed optimistic exploration methods that we are aware of. Perhaps the most interesting novelty in our method is that it radically relaxes the optimistic properties that previous methods strive for: instead of calculating estimates of the value function or the MDP model that are strictly optimistic, we only guarantee that our value estimates are optimistic in an average sense. Thus, during its runtime, our algorithm may execute several policies that do not individually satisfy any optimistic properties, even approximately. We find this property to be curious and believe that the ideas we develop to tackle such notions of “average optimism” may find other applications. We note though that our planning procedure can be used to produce optimistic policies in a stricter sense by executing several regularized value iteration steps per policy update, until the resulting optimization error vanishes. Doing so results in an improved dependency on $H$ by a factor $\sqrt{H}$ but comes at the cost of a major computational overhead.

While our algorithm is closely related to the MD-MPI method of Geist, Scherrer, and Pietquin (2019) and our proofs feature several similar steps, we remark that the purpose of our analysis is quite different from theirs, even when disregarding the optimistic adjustment we make to the Bellman operators. Taking a close look at their proofs for the special case of zero approximation errors, one can deduce bounds on our quantity of interest that are of the order $\left(H+\mathcal{H(\pi^{*}\|\pi_{K})}\right)/K$ after $K$ iterations. This is faster than what our analysis provides for approximate DP, which is due to the monotonicity of the exact Bellman operator which allows fast last-iterate convergence. The same rate appears in the analysis of regularized policy iteration methods by Agarwal et al. (2021) (see Theorem 16). Either way, all of these analyses use tools from the analysis of mirror descent first developed by Martinet (1970), Rockafellar (1976), and Nemirovski & Yudin (1983) (see also Beck & Teboulle, 2003). Note that, as the guarantees of these regularization-based methods hold on arbitrary data sequences, our regret guarantees trivially generalize to the case where the rewards change over time in a potentially adversarial fashion (as in, e.g., Even-Dar et al., 2009; Cai et al., 2020).

Another line of work that our contribution seemingly fits into is the one initiated by Liu & Su (2020) on the topic of regret minimization for discounted MDPs (see also He et al., 2021; Zhou et al., 2021). A closer look reveals that their objective is quite different from ours, in that they aim to upper bound $\sum_{t=1}^{T}\left(V^{*}\left(X_{t}\right)-V^{\pi_{t}}\left(X_{t}\right)\right)$ along the trajectory traversed by the learning agent. This notion of regret has been motivated by a formerly popular notion of “sample complexity of exploration” in discounted MDPs—we highlight Kakade (2003); Strehl et al. (2009) out of the abundant “PAC-MDP” literature on this subject. This performance measure is in fact not comparable to ours in almost any possible sense. In fact, it is easy to see that this notion may fail to capture the sample complexity of learning a good policy in a meaningful way: a policy that immediately enters a “trap” state that yields zero reward until the end of time will only incur a constant regret of order $\frac{1}{1-\gamma}$, even if there is a policy that yields a steady stream of $+1$ rewards in each round. Thus, without making stringent assumptions about the MDP that rule out such undesirable scenarios, the value of minimizing this notion of discounted regret may be questionable.

6.2 Limitations and Future Directions

On a related note, our method suffers from the limitation of requiring access to a reset action taking the agent back to the initial distribution $\nu_{0}$ at any time. In general, this is necessary to achieve our objectives. Indeed, in MDPs where all states around the initial distribution are transient, it is impossible to learn a good policy from a single stream of experience without resets since the agent only gets to visit the relevant part of the state space once. We thus believe these issues are inherent to learning in discounted MDPs.

Another limitation is that our guarantees only hold on expectation as opposed to high probability. In fact, several of our results can be strengthened to hold in this stronger sense, albeit at the cost of a more involved analysis. In particular, the only parts of our analysis that need to be changed are Lemmas 4.4 and 5.5, to deal with the randomized epoch schedule. The first of these can be handled via a martingale argument and the second by bounding the number and length of the epochs with high probability. Both of these changes are conceptually simple, but practically tedious so we omit them for clarity. On the other hand, Corollary 5.7 relies on a randomized online-to-batch conversion, and the result is stated on expectation with respect to the randomization step. Once again, this result can be strengthened to hold with high probability by running a “best-policy-selection” subroutine on the sequence of policies produced by the algorithm. This post-processing step is standard in the related literature and we omit details here to preserve clarity.

Based on our current results, generalizing our techniques to the infinite-horizon average-reward setting seems to be challenging but not impossible. The key step in our proof that requires discounting is setting the truncation level at $H=\frac{1}{1-\gamma}$, which serves the purpose of guaranteeing that our approximate Bellman operator is optimistic. In particular, the truncation level needs to be set large enough so that the inequality of Equation 3 goes through. We see no natural way to extend this condition to the undiscounted setup. We remain hopeful that this challenge can be overcome with more effort (but may potentially need some significant new ideas).

Finally, let us remark on the linear mixture MDP assumption that we have used. While arguably well-studied in the past years, this model for linear function approximation has limitations that make it rather difficult to adapt to practical scenarios. The biggest is that learning algorithms in this model need access to an oracle to evaluate sums of the form $\sum_{x^{\prime}}\psi\left(x,a,x^{\prime}\right)V\left(x^{\prime}\right)$, which can only be performed efficiently in special cases. Options include assuming that $\psi\left(x,a,\cdot\right)$ is sparse or the integral can be approximated effectively via Monte Carlo sampling. A major inconvenience that this causes in the implementation of our method is that Q-functions (and policies) cannot be represented effectively with a single low-dimensional object, so these values have to be recalculated on the fly while executing the policy, requiring excessive Monte Carlo integration in runtime. We thus wish to extend our analysis to more tractable MDP models like the model of Jin et al. (2020). While it is straightforward to implement our algorithm for linear MDPs, unfortunately the covering number of the value function class used by our algorithm appears to be too large to allow proving strong performance bounds. On a more positive note, we wish to point out that linear mixture MDPs are still a rich family of models that in general is incomparable to linear MDPs, and can subsume many interesting models—we refer to (Ayoub et al., 2020) for further discussion. We are optimistic that the limitations of our current analysis can be eventually removed and our method can be adapted to a much broader class of infinite-horizon MDPs.

Acknowledgements

G. Neu was supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 950180). Part of this work was done while the second author was visiting the Simons Institute for the Theory of Computing.

Appendix A Omitted Proofs

A.1 Technical Tools for the Proof of Lemma 4.3

Lemma A.1**.**

Let $\pi$ and $\pi^{\prime}$ be two policies, with their corresponding state-action occupancy measures being $\mu^{\pi}$ and $\mu^{\pi^{\prime}}$, and their state occupancy measures being $\nu^{\pi}$ and $\nu^{\pi^{\prime}}$. Then,

[TABLE]

Proof.

Using the chain rule of the relative entropy, we write

[TABLE]

By the Bellman flow constraints in Equation 1 and the joint convexity of the relative entropy, we bound the second term as

[TABLE]

where we also used the data-processing inequality in the last step. The proof is concluded by reordering the terms. ∎

A.2 Proof of Lemma 5.4

Let us fix $k\in\left[K\right]$, $t\in\left\{T_{k},T_{k}+1,\dots,T_{k+1}-1\right\}$, $\delta\in\left(0,1\right)$. We start by recalling the definition of the nominal transition model $\widehat{P}_{k}$ acting on functions $V$ as \bigl{(}\widehat{P}_{k}V\bigr{)}\left(x,a\right)=\bigl{\langle}\varphi_{V}\left(x,a\right),\widehat{\theta}_{k}\bigr{\rangle}, where we denoted the state-action feature map $\varphi_{V}\left(x,a\right)=\sum_{x^{\prime}\in\mathcal{X}}\psi\left(x,a,x^{\prime}\right)V\left(x^{\prime}\right)$, and the parameter $\widehat{\theta}_{k}$ can be written out as

[TABLE]

To proceed, we notice that the true transition operator acting on $V$ can be written in a similar form as

[TABLE]

where we used the definition of $\varphi_{V}$ in the last line. Proceeding further with the same expression, we write

[TABLE]

where we used the definition of $\Lambda_{T_{k}}$ and Assumption 5.3 in the last line. Comparing the expressions for $PV$ and $\widehat{P}_{k}V$, we obtain

[TABLE]

Using the Cauchy–Schwartz inequality and taking $V=V_{k}$, we get

[TABLE]

where $\xi_{k}=\left\|\sum_{i=0}^{k-1}\sum_{j=T_{i}}^{T_{i+1}-1}\varphi_{i}\left(x_{j},a_{j}\right)\left[V_{i}\left(x_{j+1}\right)-\left(PV_{i}\right)\left(x_{j},a_{j}\right)\right]\right\|_{\Lambda_{T_{k}}^{-1}}$, and $b_{k}=\lambda\left\|\theta\right\|_{\Lambda_{T_{k}}^{-1}}$. The second term can be easily bounded as $\left|b_{k}\right|\leq\sqrt{\lambda}\left\|\theta\right\|_{2}\leq\sqrt{\lambda}B$, using that $\lambda_{\text{min}}\left(\Lambda_{T_{k}}\right)\geq\lambda$ and the boundedness of the features.

For the first term, observe that $V_{i}\left(x_{j+1}\right)-\left(PV_{i}\right)\left(x_{j},a_{j}\right)$ forms a martingale difference sequence, with increments bounded in $\left[-H,H\right]$ by the truncation made in the algorithm. Additionally, the feature vectors are bounded as $\left\|\varphi_{i}\left(x_{j},a_{j}\right)\right\|_{2}\leq BH$ and the true parameter as $\left\|\theta\right\|_{2}\leq B$ by Assumption 5.3. Therefore, we can apply the self-normalized concentration result in Theorem C.2 (stated in Appendix C.2), which guarantees that with probability at least $1-\delta$, the following bound holds for all $k\in\left[K\right]$:

[TABLE]

The determinants appearing in the bound can be further upper bounded by using that $\det\left(\lambda I\right)=\lambda^{d}$ and

[TABLE]

where in the last step we used $T_{k}\leq T$. We plug this back in the upper-bound on $\xi_{k}$ to obtain the bound

[TABLE]

Putting everything together, we have verified that, for all $k\in\left[K\right]$,

[TABLE]

holds with probability at least $1-\delta$, where we have defined $\beta$ as

[TABLE]

This concludes the proof. ∎

A.3 Proof of Lemma 4.4

For the sake of this proof, we slightly update our notation for $\mathcal{T}_{k}$ by setting $\mathcal{T}_{K\left(T\right)}=\left\{T_{K\left(T\right)},T_{K\left(T\right)}+1,\dots,T^{+}\right\}$. We will use $\mathcal{F}_{k-1}$ to denote the filtration generated by the observations up to the end of epoch $k-1$, and $L_{k}$ to denote the length of epoch $k$. We start by rewriting the sum of exploration bonuses up to step $T^{+}$ as

[TABLE]

By virtue of the definition of $T^{+}$, all epochs are of geometric length with mean $\frac{1}{1-\gamma}$. Now, let us consider a fixed epoch $k$ and define the auxiliary infinite sequence of state-action pairs $X_{k,0},A_{k,0},X_{k,1},A_{k,1},\dots$ that is generated independently from the realized sample trajectory $\left(X_{t},A_{t}\right)_{t\in\mathcal{T}_{k}}$ given $\mathcal{F}_{k-1}$ as follows. The initial state $X_{k,0}$ is drawn from $\nu_{0}$, and then subsequently for each $i=0,1,\dots$, the actions are drawn as $A_{k,i}\sim\pi_{k}\left(\cdot\middle|X_{k,i}\right)$ and follow-up states are drawn as $X_{k,i+1}\sim P\left(\cdot\middle|X_{k,i},A_{k,i}\right)$. Recalling the notational convention that $\textup{CB}_{t}=\textup{CB}_{k}$ for all $t\in\mathcal{T}_{k}$, we observe that for any $k$, we have

[TABLE]

where in the third line we have observed that $L_{k}$ follows a geometric law with parameter $1-\gamma$, and is independent of $\left(X_{k,i},A_{k,i}\right)_{i}$. In the fourth line we introduced the notation $u_{k,i}$ to denote the joint distribution of $X_{k,i},A_{k,i}$ given $\mathcal{F}_{k-1}$ and noticed that the discounted sum of these distributions exactly matches the definition of the occupancy measure $\mu^{\pi_{k}}$ up to the normalization constant $\left(1-\gamma\right)$, and finally concluded by observing that $\mathbb{E}\left[\left.L_{k}\right|\mathcal{F}_{k-1}\right]=\frac{1}{1-\gamma}$.

The proof is completed by summing up over all epochs, taking marginal expectations, and noticing that

[TABLE]

∎

A.4 Proof of Lemma 5.5

The proof is based on a classic “pigeonhole” argument often called the “elliptical potential lemma” (e.g., Lemma 19.4 in Lattimore & Szepesvári, 2020, or Section 11.7 in Cesa-Bianchi & Lugosi, 2006, but see also Lai et al., 1979; Lai & Wei, 1982). The main challenge of adapting this result to our setting is accounting for the randomized epoch schedule. Another subtle difficulty comes from the fact that Lemma 4.3 only bounds the total regret as opposed to the instantaneous regrets in each round, which necessitates arguments that are slightly more involved than what is commonly seen in closely related work.

As for the actual proof, we start by introducing some useful notation that we will use throughout the proof. For $t\in\left[T\right]$, we use $k_{t}$ to denote the index of the epoch that $t$ belongs to. For simplicity, for all $k$ and $t$, we will write $\varphi_{k,t}=\varphi_{k}\left(X_{t},A_{t}\right)$, $\Lambda_{k}=\Lambda_{T_{k}}$. We also define $\mathcal{N}\left(T\right)=\left\{t\in\left[T\right]:\left\|\varphi_{k_{t},t}\right\|_{\Lambda_{k_{t}}^{-1}}\geq 1\right\}$ as the set of “bad” time indices where state-action pairs with large feature norms are observed, and $\mathcal{E}\left(T\right)=\left\{k\in\left[K\left(T\right)\right]:\exists t\in\mathcal{T}_{k},\left\|\varphi_{k,t}\right\|_{\Lambda_{k}^{-1}}\geq 1\right\}$ be the set of epochs containing at least one bad time index. Using these definitions, we rewrite the sum of exploration bonuses as follows:

[TABLE]

where in the last step we used the definition of $\mathcal{E}(T)$. We treat the first term separately in Lemma A.3, stated after this proof. This gives the following bound:

[TABLE]

Thus, we can focus on the second term in the right hand side. This term can be upper-bounded using the Cauchy–Schwarz inequality as

[TABLE]

To proceed, we use the inequality $\left(x\wedge\left|\mathcal{T}_{k}\right|\right)\leq\frac{\left|\mathcal{T}_{k}\right|}{\log\left(\left|\mathcal{T}_{k}\right|+1\right)}\log\left(1+x\right)$ that is valid for all $x\geq 0$. Setting $C_{k}=\frac{\left|\mathcal{T}_{k}\right|}{\log\left(\left|\mathcal{T}_{k}\right|+1\right)}$, this gives

[TABLE]

The sum is handled separately in Lemma A.2 stated and proved right after this proof. Putting the result together with our previous calculations, we get

[TABLE]

The only random quantity left in the upper-bound is the maximum over $C_{k}$. By concavity of the square-root function and Jensen’s inequality, we get

[TABLE]

which we further upper bound by using Lemma A.4 as

[TABLE]

We put together the two terms, and plug in the definition of $\beta$ to get

[TABLE]

where the two factors are defined as

[TABLE]

The proof is then concluded by observing that $C_{1}\left(T\right)+C_{2}\left(T\right)=\mathcal{O}\left(BH^{3/2}d\log\left(T\right)^{3/2}\right)$. ∎

Lemma A.2**.**

Following the same notations that in Section A.4

[TABLE]

Proof.

We will follow the steps of the proof of Lemma 19.4 in Lattimore & Szepesvári (2020). First, notice that $\Lambda_{k}$ can be written as

[TABLE]

Taking the determinant of the above matrix, we get

[TABLE]

Now, taking logarithms on both sides and using the concavity of $\log\det$, we obtain

[TABLE]

where the inequality is Jensen’s, and the final step follows from using the equality $\det(I+vv^{\mathsf{\scriptscriptstyle T}})=(1+\left\|v\right\|_{2}^{2})$ that holds for any $v\in^{d}$. Summing up for $k$ gives

[TABLE]

and furthermore by the trace-determinant inequality we have

[TABLE]

Finally, the trace can be bounded as

[TABLE]

Plugging this back into the previous inequality proves the claim. ∎

Lemma A.3**.**

The number of epochs that contain a feature vector with a norm larger than one is bounded as

[TABLE]

A simpler version of this statement is given as Exercise 19.3 in Lattimore & Szepesvári (2020), and our proof below drew inspiration from the proof of Lemma 19 in (Ouhamma et al., 2022). We only have to deal with the challenge of randomized epoch schedules, which we do by similar arguments as in the proof of Lemma 5.5 above.

Proof.

Let $k\in\left[K\left(T\right)\right]$. We define $G_{0}=\lambda I$ and $G_{k+1}=G_{k}+\sum_{t\in\mathcal{T}_{k}}\varphi_{k,t}\varphi_{k,t}^{\mathsf{\scriptscriptstyle T}}\mathbb{I}_{\left\{t\in\mathcal{N}\left(T\right)\right\}}$. We have the following decomposition:

[TABLE]

Therefore, taking the log-determinant on both sides, we obtain

[TABLE]

If $\mathcal{T}_{k}\cap\mathcal{N}\left(T\right)=\emptyset$, or equivalently if $k\notin\mathcal{E}\left(T\right)$ (i.e., there is no “bad” state-action pair in the epoch $k$), the second term in the right-hand side is zero. Hence, summing over $k\in\left[\mathcal{E}\left(T\right)\right]$, we get

[TABLE]

Using the concavity of $\log\det$, Jensen’s inequality gives us

[TABLE]

where the equality follows from the fact that $\det(I+vv^{\mathsf{\scriptscriptstyle T}})=(1+\left\|v\right\|_{2}^{2})$ that holds for any $v\in^{d}$. Then, we notice that $G_{k}^{-1}\succeq\Lambda_{k}^{-1}$, and thus we can further bound this expression as

[TABLE]

For $k\in\mathcal{E}\left(T\right)$, $t\in\mathcal{T}_{k}\cap\mathcal{N}\left(T\right)$, we have $\left|\mathcal{T}_{k}\cap\mathcal{N}\left(T\right)\right|\geq 1$, and $\left\|\varphi_{k,t}\right\|_{\Lambda_{k}^{-1}}\geq 1$ by definition of $\mathcal{N}\left(T\right)$. This implies that

[TABLE]

Thus, we have

[TABLE]

Finally, the trace can be bounded as

[TABLE]

The proof is concluded by putting this bound together with the previous inequality. ∎

Lemma A.4**.**

The random variables $\left\{C_{k}\right\}_{k}$, defined for all $k$ by $C_{k}=\frac{\left|\mathcal{T}_{k}\right|}{\log\left(1+\left|\mathcal{T}_{k}\right|\right)}$ where $\left|\mathcal{T}_{k}\right|$ is a geometric random variable with parameter $1-\gamma$, satify the following

[TABLE]

Proof.

First, notice that $\log\left(1+\left|\mathcal{T}_{k}\right|\right)\geq\log 2$ so that $C_{k}=\frac{\left|\mathcal{T}_{k}\right|}{\log\left(1+\left|\mathcal{T}_{k}\right|\right)}\leq\frac{\left|\mathcal{T}_{k}\right|}{\log 2}$. Next, using the fact that the number of epochs is at most $T$, and observing that each $\left|\mathcal{T}_{k}\right|$ is geometrically distributed with parameter $1-\gamma$, we can bound $\max_{k}\left|\mathcal{T}_{k}\right|$ by a maximum over $T$ independent geometric random variables $Z_{1},\dots,Z_{T}$ with parameter $1-\gamma$:

[TABLE]

where we have used the formula for the geometric sum in the last step. Now, setting $k=\left\lceil\frac{\log T}{1-\gamma}\right\rceil$, we get

[TABLE]

where in the second line we have used the inequality $\frac{\log\gamma}{1-\gamma}\leq-1$ that holds for all $\gamma\in\left(0,1\right)$. The proof is concluded by using that $\log 2>\frac{1}{2}$ and combining the above bound with the bound relating $C_{k}$ to $\left|\mathcal{T}_{k}\right|$. ∎

Appendix B Applications: Algorithm Specifications

For clarity, we provide the complete algorithms in the context of tabular and linear mixture MDPs. The highlighted parts correspond to the instantiations of the functions TRANSITION-ESTIMATE, BONUS, and ADD from Algorithm 1.

B.1 Tabular MDPs

In tabular MDPs, we use the maximum likelihood estimates to compute $\widehat{P}_{k}$ and the classical count-based bonuses. Therefore, we only need to store and update the counts $N_{t}$ and $N_{t}^{\prime}$ when interacting with the environment.

B.2 Linear Mixture MDPs

In linear mixture MDPs, $\widehat{P}_{k}$ is computed via a least-squares regression, and we use elliptical bonuses for $\textup{CB}_{k}$. Thus, we only need to store and update the empirical covariance matrix $\Lambda_{t}$ and the vector $b_{t}$ when interacting with the environment.

Appendix C Standard Results

C.1 Softmax Policies and Value Functions

In this section, we recall a range of standard facts relating the softmax policies our algorithm uses and the associated value functions. These can be found in numerous papers, textbooks, and lecture notes—for concreteness, see Section 28.1 in Lattimore & Szepesvári, 2020 as an example.

Lemma C.1**.**

Let $\left\{V_{k}\right\}_{k\in\left[K\right]}$, $\left\{\pi_{k}\right\}_{k\in\left[K\right]}$, and $\left\{Q_{k}\right\}_{k\in\left[K\right]}$ be the sequences of functions defined in Algorithm 1. Then, the following equalities are satisfied for all $k\in[K]$ and $x\in\mathcal{X}$:

[TABLE]

Furthermore, for all $k\in[K]$ and $x\in\mathcal{X}$, we have

[TABLE]

Proof.

First, we show that the maximum indeed takes the form claimed in the main paper and that the maximizer is given by a softmax policy. For simplicity, we drop the indices for now and consider the optimization problem

[TABLE]

where $Q\in\mathbb{R}^{\mathcal{A}}$, and $p^{\prime}\in\Delta\left(\mathcal{A}\right)$. As the probability simplex is compact and $\left(p\mapsto\left\langle p,Q\right\rangle-\frac{1}{\eta}\mathcal{D}_{\text{KL}}\left(p\|p^{\prime}\right)\right)$ is continuous, the supremum is attained at some $p^{*}\in\Delta\left(\mathcal{A}\right)$. The Lagrangian function of this optimization problem is given for all $p\in\mathbb{R}_{+}^{\mathcal{A}}$ and $\alpha\in\mathbb{R}$ as

[TABLE]

Its partial derivative with respect to the primal variable $p(a)$ is

[TABLE]

Setting it to zero gives us the expression

[TABLE]

Then, we use the constraint on $p^{*}$ to find the value of $\alpha$. In particular, $\left\langle p^{*},\mathbf{1}\right\rangle=1$ implies

[TABLE]

from which we deduce that

[TABLE]

Denoting $V^{*}=\frac{1}{\eta}\log\left(\sum_{a\in\mathcal{A}}p^{\prime}\left(a\right)\exp\left[\eta Q\left(a\right)\right]\right)$, we plug back the expression of $\alpha$ into $p^{*}$:

[TABLE]

From this, we can directly express the relative entropy between $p^{*}$ and $p^{\prime}$ as

[TABLE]

so that we can write

[TABLE]

The first statement of the lemma then follows from applying this result to $Q=Q_{k}\left(x,\cdot\right)$ and $p^{\prime}=\pi_{k-1}\left(\cdot\middle|x\right)$, for $k\in\left[K\right]$, $x\in\mathcal{X}$. That is, for any state-action pair $\left(x,a\right)\in\mathcal{X}\times\mathcal{A}$ and $k\geq 1$, denoting the maximum $V_{k}$ and the maximizer $\pi_{k}$, we have that the following expressions are equivalent to the ones given in the statement of the lemma:

[TABLE]

For the second statement, we start by denoting $\bar{V}_{k}=\sum_{i=1}^{k}V_{i}$ and $\bar{Q}_{k}=\sum_{i=1}^{k}Q_{i}$, for $k\geq 1$, and show by induction that, for $x\in\mathcal{X}$, the following holds

[TABLE]

Let $x\in\mathcal{X}$. The case $k=1$ follows immediately from the previous statement with $Q=Q_{1}\left(x,\cdot\right)$ and $p^{\prime}=\pi_{0}\left(\cdot\middle|x\right)$. Assume the previous equation holds at $k$. Using the first statement with $Q=Q_{k+1}\left(x,\cdot\right)$ and $p^{\prime}=\pi_{k}\left(\cdot\middle|x\right)$ we have, for $a\in\mathcal{A}$, $\pi_{k+1}\left(a\middle|x\right)=\pi_{k}\left(a\middle|x\right)e^{\eta\left[Q_{k+1}\left(x,a\right)-V_{k+1}\left(x\right)\right]}$. Applying the inductive hypothesis, it gives

[TABLE]

which finishes the induction. Then, we move on to the actual statement. We have

[TABLE]

Therefore, by definition of $\bar{V}_{k-1}$,

[TABLE]

which concludes the proof. ∎

C.2 A Self-Normalized Tail Inequality

Theorem C.2** (Theorem 14.7 in de la Peña et al. (2009), Theorem 2 in Abbasi-Yadkori et al. (2011)).**

Let $\left\{\eta_{t}\right\}_{t=1}^{\infty}$ be a real-valued stochastic process with corresponding filtration $\left\{\mathcal{F}_{t}\right\}_{t=0}^{\infty}$. Let $\eta_{t}|\mathcal{F}_{t-1}$ be zero-mean and $\sigma$-subGaussian; i.e. $\mathbb{E}\left[\eta_{t}\middle|\mathcal{F}_{t-1}\right]=0$, and

[TABLE]

Let $\left\{\varphi_{t}\right\}_{t=0}^{\infty}$ be an $\mathbb{R}^{d}$-valued stochastic process where $\varphi_{t}$ is $\mathcal{F}_{t-1}$-measurable. Assume $\Lambda_{0}$ is a $d\times d$ positive definite matrix, and let $\Lambda_{t}=\Lambda_{0}+\sum_{s=1}^{t}\varphi_{s}\varphi_{s}^{\mathsf{\scriptscriptstyle T}}$. Then, for any $\delta>0$, with probability at least $1-\delta$, we have for all $t\geq 0$,

[TABLE]