A Tractable Algorithm For Finite-Horizon Continuous Reinforcement Learning

TL;DR

This paper introduces a tractable optimistic value iteration algorithm for finite-horizon continuous reinforcement learning, establishes regret lower bounds, analyzes discretization errors, and validates findings through experiments.

Contribution

It presents a new algorithm for finite-horizon continuous RL, derives regret lower bounds, and analyzes discretization errors under H"{o}lder continuity assumptions.

Findings

Proposed a tractable optimistic value iteration algorithm.

Established a regret lower bound of (T^{2/3}) for discretized state spaces.

Analyzed discretization error bounds under H"{o}lder continuity.

Abstract

We consider the finite horizon continuous reinforcement learning problem. Our contribution is three-fold. First,we give a tractable algorithm based on optimistic value iteration for the problem. Next,we give a lower bound on regret of order $\Omega(T^{2/3})$ for any algorithm discretizes the state space, improving the previous regret bound of $\Omega(T^{1/2})$ of Ortner and Ryabko \cite{contrl} for the same problem. Next,under the assumption that the rewards and transitions are H\"{o}lder Continuous we show that the upper bound on the discretization error is $const.Ln^{-\alpha}T$. Finally,we give some simple experiments to validate our propositions.

Equations

$$\big{|}r(s,a)-r(s^{\prime},a)\big{|}\leq L|s-s^{\prime}|^{\alpha}.$$

$$\big{\|}p(\cdot|s,a)-p(\cdot|s^{\prime},a)\big{\|}_{1}\leq L|s-s^{\prime}|^{\alpha}.$$

$$V^{\pi_{k}}_{h}(s)=\mathds{E}_{\pi_{k}}\bigg{(}\sum_{i=h}^{H}r(x_{k,i},\pi_{k}(x_{k,i},i))|x_{k,h}=s\bigg{)}.$$

$$\mbox{Regret}(K)=\sum_{k=1}^{K}\big{(}V_{1}^{*}(x_{k,1})-V_{1}^{\pi_{k}}(x_{k,1})\big{)}.$$

$$p^{agg}(I_j|s,a)={\int }_{I_j}p(d{s}^{\prime }|s,a)$$

$$r^{agg}(I_j,a)=n.{\int }_{I_j}r(s,a)ds.$$

$${\pi }_k(s,i)={\pi }_k^{agg}(I(s),i)$$

$$V^{agg}_{\pi_{k,h}}(I)=\mathds{E}_{\pi^{agg}_{k}}\bigg{(}\sum_{i=h}^{H}r^{agg}(I_{k,i},\pi^{agg}_{k}(I_{k,i},i))|I_{k,h}=I\bigg{)}.$$

$$P^{agg}(I|I^{\prime },a)=n.{\int }_{I^{\prime }}{p}^{agg}(I|t,a)dt$$

$$b(x,a)=\sqrt{\frac{8LVa{r}_{Y\sim {\hat{P}}_k^{agg}(.|x,a)}(V_{k,h+1}^{agg}(Y))}{N_k(x,a)}}$$

$$+\frac{14HL}{3N_k(x,a)}$$

$$+\sqrt{\frac{8{\sum }_y{\hat{P}}_k^{agg}(y|x,a)[\min (\frac{100H^3{S}^{2}A{L}^2}{N_{k,h+1}^{\prime }(y)},H^2)]}{N_k(x,a)}}$$

$$\text{mbox}Regret(K)=\sum _{k=1}^K$$

$$+V_{{\pi }_{k,1}}^{agg}(I_{k,1})-V_1^{{\pi }_k}(x_{k,1})]$$

$$\displaystyle+\underbrace{[V^{agg^{*}}_{1}(I_{k,1})-V^{agg}_{\pi_{k,1}}(I_{k,1})]}_{\Delta_{agg}}\big{)}$$

$${\Delta }_{error}=\sum _{k=1}^K$$

$$+V_{{\pi }_{k,1}}^{agg}(I_{k,1})-V_1^{{\pi }_k}(x_{k,1})]$$

$$q_1^k$$

$$q_2^k$$

$${\Delta }_{error}$$

$$V_1^{\ast }(x_{k,1})$$

$$(\text{mbox}Chapter4of\text{cite}[cite][\text{@@bibref}hernandez2012discrete])$$

$$V_1^{ag{g}^{\ast }}(I_{k,1})=\underset{a\in A}{\max }$$

$$\displaystyle+\sum_{j=1}^{n}P^{agg}(I_{j}|I_{k,1},a)V^{agg^{*}}_{2}(I_{j})\big{)}$$

$$(\text{mbox}BellmanEquation)$$

$$\text{mbox}Usingtheproperty|\max f$$

$$\displaystyle|q_{1}^{k}|\leq\max_{a\in A}\Big{|}\big{[}r(x_{k,1},a)$$

$$\displaystyle+\big{[}\int_{S}p(ds^{\prime}|x_{k,1},a)$$

$$\displaystyle-\sum_{j=1}^{n}P^{agg}(I_{j}|I_{k,1},a)V^{agg^{*}}_{2}(I_{j})\big{]}\Big{|}$$

$$\text{mbox}Let{l}_1^k$$

$$l_2^k$$

$$\displaystyle-\sum_{j=1}^{n}P^{agg}(I_{j}|I_{k,1},a^{\prime})V^{agg^{*}}_{2}(I_{j})\Big{|}$$

$$\max |f+g|\le \max |f|+\max |g|$$

$$|q_1^k|$$

$$\displaystyle\big{|}r(x_{k,1},a)$$

$$\displaystyle\leq\bigg{|}r(x_{k,1},a)-n.\bigg{(}\sum_{i=1}^{n}m_{i}\Delta x_{i}\bigg{)}\bigg{|}$$

$$\displaystyle=\bigg{|}r(x_{k,1},a)-n.\bigg{(}\sum_{i=1}^{n}r(s^{\prime}_{i},a)\frac{1}{n^{2}}\bigg{)}\bigg{|}$$

$$\displaystyle\leq\frac{1}{n}.\sum_{i=1}^{n}\big{|}r(x_{k,1},a)-r(s^{\prime}_{i},a)\big{|}\leq Ln^{-\alpha}.$$

$$\displaystyle\leq\big{|}H\sum_{j=1}^{n}\int_{I_{j}}p(ds^{\prime}|x_{k,1},a)-\sum_{j=1}^{n}P^{agg}(I_{j}|I_{k,1},a)V^{agg^{*}}_{2}(I_{j})\big{|}$$

$$\in [0,1]$$

$$\displaystyle\leq\sum_{j=1}^{n}\big{|}H\int_{I_{j}}p(ds^{\prime}|x_{k,1},a)-P^{agg}(I_{j}|I_{k,1},a)V^{agg^{*}}_{2}(I_{j})\big{|}$$

Full text

I Introduction

In most reinforcement learning algorithms the size of the state and action spaces are assumed to be finite. But many real-world applications have continuous state or action spaces. Though the case of continuous state space has been considered before like in Ortner and Ryabko [1] and Lakshmanan et al. [2] there is no tractable algorithm whose regret has been analyzed.In this work we consider a simpler finite-horizon setting and give a tractable algorithm with near-optimal regret bound.

There have been many assumptions on the rewards and transition function considered before like deterministic transitions in [3] or transition functions that are linear in state and action [4], [5], [6] and [7]. Kakade et al. [8] have considered more general setting of PAC learning in RL with metric state spaces. Another result is by Osband and Roy [9] where bounds on expected regret was derived when reward and transition functions belonged to a class of functions characterized by parameters like eluder dimension and Kolmogorov dimension. In this paper we consider the most general assumption that the reward function and the transition functions are Hölder continuous like the papers [1] and [2]. We derive our bound on the discretization error without any further assumptions. By assuming unbiased sampling, we also derive an upper bound on the regret which is $O(T^{2/3})$ when both transition probabilities and reward function are Lipschitz continuous.

In recent work, Azar et al. [10] have proposed a new algorithm Optimistic Value Iteration (UCBVI). This is different from the UCRL algorithm in Jacksh et al [11] upon which the algorithms of Ortner and Ryabko [1] was based. For episodic problems, it is known that UCBVI algorithm has better regret bounds. We extend this UCBVI algorithm for continuous state space problems using the state space aggregation used in [1]. It should also be noted that both the algorithms in [1] and [2] are not tractable. Though we have considered an easier episodic setting, our algorithm is tractable. And we show that the discretization error(discretizing the state space to intervals) is bounded above by $const.Ln^{-\alpha}T$ under the assumptions that rewards and transitions are Hölder Continuous.

For infinite horizon setting, in the one-dimensional case with Lipschitz rewards and transition functions the bound by Ortner and Ryabko is of order $\tilde{O}(T^{3/4})$ in [1]. If the transition functions are in addition smooth, a regret bound of $O(T^{2/3})$ is shown in [2].

The best known lower bound for Lipschitz reward and general transition function is by Ortner and Ryabko [1] which is order $\Omega(T^{1/2})$. We improve upon this by giving a lower bound of $\Omega(T^{2/3})$, matching the upper bound for algorithms that discretizes the state-space. We have also implemented the algorithm and give experimental results on one and two dimensional setting. The empirical performance is seen to match with the theoretical results.

II Terminology and Problem Definition

We define Markov Decision Process (MDP) by state space $S$, action space $A$, reward function $r:S\times A\rightarrow[0,1]$ and transition probability $P:S\times A\times S\rightarrow[0,1]$. We consider continuous state space $[0,1]^{d}$ and finite action MDP. For simplicity we derive results for the one dimensional state space, they can be easily generalized to higher dimensional state spaces.The random rewards given state $s$ and action $a$ are bounded in $[0,1]$ with mean $r(s,a)$. The probability of going to state $s^{\prime}$ given state $s$ and action $a$ is given by the transition probability $p(s^{\prime}|s,a)$. We make the following assumptions like in [1], [2]. This guarantees that rewards and transitions are close in close states.

Assumption 1**.**

There are $L,\,\alpha>0$ such that for any two states $s,\,s^{\prime}$ and all actions a,

[TABLE]

Assumption 2**.**

There are $L,\,\alpha>0$ such that for any two states $s,\,s^{\prime}$ and all actions a,

[TABLE]

For simplicity we assume $L,\,\alpha$ are same for both assumptions.

We consider the finite horizon setting where the agent interacts with the environment in $H$ number of steps per episode. We denote by $[n]$ the set $\{i\in\mathbb{N},1\leq i\leq n\}$). The policy during an episode is expressed as a mapping $\pi:S\times[H]\to A$. Let $x_{k,h}$ denote the state in the step $h$ of the episode $k$ and $\pi_{k}$ denote the policy for the episode $k$. The value function of each state in $k^{th}$ episode from step $h$ by following a policy $\pi_{k}$ is defined by

[TABLE]

The optimal value function is defined as $V_{h}^{*}(x)=\sup\limits_{\pi}V_{h}^{\pi}(x)$ for all $x\in S$ and $h\geq 1$. The performance of the algorithm is measured according to regret incurred in all the episodes as given by

[TABLE]

We discretize the continuous state space into intervals of length $1/n$ like in [1]. Let $I(s)$ denote interval of the state $s\in S$. We define aggregate rewards and aggregate transition probabilities with respect to $\{I_{1},I_{2},\ldots,I_{n}\}$ as

[TABLE]

[TABLE]

Here $r^{agg}(I_{j},a)$ can be interpreted as the mean reward in the interval $I_{j}$. The algorithm treats each interval as single state and the aggregate policy is $\pi_{k}^{agg}(I(s),i)$. The aggregated value function following this policy is $V^{agg}_{\pi_{k},h}(I_{j})$.At any state $s\in S$ and and step $i\in[H]$,the mapping between the policies are given by

[TABLE]

[TABLE]

For the discretised MDP, the transitions are between intervals,so we define $P^{agg}(I|I^{\prime},a)$ as the average of $p^{agg}(I|t,a)$ in $I^{\prime}$

[TABLE]

III Algorithm

In UCRL [11] based algorithms confidence sets are built around rewards and transition probabilities. But here we build the confidence set around optimal value function $V^{*}_{h}$ of the discretized MDP as in Azar et al ([10]). The algorithm proceeds similar to UCBVI in [10]. The only difference is that here we use aggregated rewards, transitions probabilities and value functions. The bonus (Algorithm 2) which is used in calculating the Q-Values is built from the empirical variance of the estimated next values. This relies on the Bernstein-Freedman’s concentration inequality for building the concentration sets.

In UCRL based algorithms [11] we need to find the optimistic MDP and optimal policy for that MDP. This step is not tractable when the state space is continuous as we need knowledge of the bias span $H$ (section 3 of [1]). Though this is not needed for finite-horizon problems, to the best of our knowledge, UCCRL algorithm is not computationally tractable. We note that in our algorithm which is based on UCBVI, this step is not needed. Briefly the algorithm consists of the following parts/sections.

Initialization part which also includes the discretizing the continuous MDP based on the input parameters. This step is executed only once. 2. 2.

The next part consists of an iterative flow of three processes.(The number of iterations is equal to the number of episodes $K$)

• •

Estimating the transition probabilities based on the history till that iterate.

• •

Finding the Q values using a modified bellman operator(which includes bonus) according to the current transition probabilities.This part can be considered as a simple Dynamic Programming(DP) problem for finding Q values.

• •

Execute the current(discrete) policy(according to the Q values found) and record the feedback given by the environment.

III-A Bonus

This algorithm follows the heuristic principle known as optimism in the face of uncertainty. The algorithms in this paradigm employ optimism to guide exploration. In simple words, the algorithm assigns higher bonus value to that action(given a state/interval) which it is most uncertain about. This encourages the exploration of that action next time when it visits the same state again. Indirectly as the number of times the action chosen increases the algorithm would become more and more certain about it. Given a (state/interval,action) pair, the bonus in the algorithm depends on the variance of value function of the next possible states. Larger the variance,larger the uncertainty of taking that action.

Let us understand this using a simple example. Consider a discretized MDP with 4 possible actions. Given a particular interval $I_{k}$, let the Q values and the corresponding bonus found by the algorithm be equal to those given in the figure 1. In the case of not using bonus, the best action would be $a1$ because $(I_{k},a1)$ has the highest Q-value in $\{(I_{k},a_{i})|i=1,2,3,4\}$. But when bonus is considered, the action $a4$ is the most likely to be chosen. According to the bonus, the decreasing order of uncertainties of actions are $a4,a2,a1,a3$. The combined effect of Q value and the bonus will lead to the exploration of action $a4$. Due to this exploration of action $a4$, the uncertainty about choosing $a4$ reduces.

IV Results

The regret analysis of UCBVI-CRL is straight forward. The regret of the continuous MDP upto K episodes is defined in equation (4). Now add and subtract terms $V^{agg^{*}}_{1}(I_{k,1})$, $V^{agg}_{\pi_{k,1}}(I_{j})$ to this to get,

[TABLE]

Here $\Delta_{agg}$ is the regret of the discretized MDP and $\Delta_{error}$ is the error which is a consequence of discretizing the state space.

IV-A Bounding $\Delta_{error}$

Lemma 1**.**

Using the discretization technique mentioned above, $\Delta_{error}$ is bounded above by $\bm{const.Ln^{-\alpha}T}$.

Proof.

Let us split the terms in $\Delta_{error}$ into two parts as

[TABLE]

Consider $|q_{1}^{k}|$

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

For any action $a\in A$, $l_{1}^{k}$ is bounded by

[TABLE]

The first inequality is obtained by replacing the integral with lower Riemann sum by dividing $I_{k,1}$ into n intervals each of length $\Delta x_{i}=\frac{1}{n^{2}}$ and $m_{i}$ being the infimum in the sub-interval. The second inequality follows since function $r$ is continuous w.r.t $s$ and infimum is the minimum. The last inequality follows from the assumption (1) and length of $I_{j}=\frac{1}{n}$.

For any action $a\in A$, $l_{2}^{k}$ is bounded by

[TABLE]

[TABLE]

From (9), replacing the integral with lower Riemann sum with n sub intervals and using the property of continuity that infimum of a function is equal to minimum in that closed interval.

[TABLE]

Now we have,

[TABLE]

Hence $|q_{2}^{k}|$ can be bound by bounding the difference of the rewards inside the expectation which is similar to bounding $l_{1}^{k}$. For any action $a$ we have,

[TABLE]

Now $\Delta_{error}$ is bounded by

[TABLE]

∎

V Additional Result: Upper Bound on Regret

We derived a tighter upper bound on the regret for continuous case(when using the proposed algorithm) when the transition estimates found by the algorithm are unbiased estimates of (9).The below theorem states that.

Theorem 1**.**

Let M be an MDP with continuous state space $[0,1]$, $A$ actions, known reward function $r$ and unknown transitions satisfying assumptions 1,2 and the transition estimates are unbiased estimates of (9). Then with probability $1-\delta$, the regret of UCVBI-CRL after T steps is upper bounded by

[TABLE]

where $L^{\prime}=ln(5HnAT/\delta)$.

Proof.

The regret in the continuous setting can be split into $\Delta_{error}$ and $\Delta_{agg}$. The bound on $\Delta_{error}$ comes from Lemma 1 and bound on $\Delta_{agg}$ is from Theorem 2 of [10].∎

Corollay 1**.**

We set $n=T^{x}$. It can be seen that when $\alpha\in[1/k,1]$, setting $x<\frac{k}{2k+1}$ first term dominates while the third term $2500H^{2}n^{2}AL^{2}$ dominates when $x>=\frac{k}{2k+1}$. Thus the optimal regret of $O(L^{2}H^{2}AT^{2k/2k+1})$ is obtained by setting $x=\frac{k}{2k+1}$. This is better than the regret bound of order $T^{\frac{2k+1}{2k+2}}$ for the same in [1]. And in the Lipschitz case i.e., when $\alpha=1$ we have regret of order $O(T^{2/3})$.

Remark 1**.**

*The dependence on $H$, $L$ and $A$ is not optimal. If the algorithms of [1] and [2] are adapted for finite-horizon setting We again wish to emphasize the fact that our algorithm is tractable unlike the algorithms in the other two papers. *

Remark 2**.**

The regret bound in [2] is also of order $\tilde{O}(T^{2/3})$ for the infinite horizon problem in the Lipschitz case. But there is an additional parameter $\beta$ which depends on the smoothness of the transition function. And only in the asymptotic case when $\beta\rightarrow\infty$ the regret of $\tilde{O}(T^{2/3})$ is attained.

Remark 3**.**

For $d$-dimensional state space we have $n^{d}$ states in the last three terms of equation (6). So setting $n=T^{\frac{k}{2kd+1}}$ we get a regret of $\tilde{O}(T^{\frac{2kd}{2kd+1}})$ which is better than the bound of $\tilde{O}(T^{\frac{2kd+1}{2kd+2}})$ in [1] for the case $\alpha\in[1/k,1]$.

VI Lower Bounds

We have the following theorem giving the lower bound.

Theorem 2**.**

For any algorithm $\mathcal{A}$ using state-space discretization, any natural numbers $A$ and $T\geq A$, the expected regret for $\mathcal{A}$ after $T$ timesteps for the MDP $\mathcal{M}$ with state $S=[0,1]$, actions $A$ defined above is

[TABLE]

Proof.

We note that any algorithm using state-space discretization also incurs a regret by choosing a wrong action due to discretization error. Hence we have

Regret in the discretized MDP 2. 2.

Regret due to the discretization error

The total regret is atleast maximum of these two. The first term is of order $\sqrt{nHAT}$ (See [11]). Now consider the MDP shown in figure 2 with two actions in each state.

Let the algorithm discretize the state into $n$ intervals.Let us assume that there are two sub-intevals in each interval and the optimal action in each sub-interval is different. Assume for simplicity that all intervals are of equal size. Otherwise divide each interval into sub-intervals of size half of the least interval size and assume constant reward for the remaining length of the intervals.

The mean rewards for actions are as shown in the figure 2, i.e., they are fixed for a state. It can be seen that it is not possible to determine the optimal action for both these sub-intervals simultaneously. And the regret obtained due to discretization is of order $L.\frac{1}{2n}$ as the the rewards are Lipschitz continuous. Since this regret is incurred on an average for atleast half of the states visited $T/2$. The total regret is of order $L.\frac{T}{4n}$. Balancing this and the regret in the discretized MDP by taking $n=T^{1/3}$ we get a regret of order $\Omega(L.\sqrt{HA}T^{2/3})$. ∎

Remark 4**.**

We note that algorithms like value iteration and non-stationary value iteration algorithms (Chapter 3 of [13]) which do not use state-space discretization cannot be used here as the probability transition function is not known.

Remark 5**.**

It is straight forward to show a regret bound of $\Omega(L.\sqrt{H_{sp}A}T^{2/3})$ for infinite horizon setting where $H_{sp}$ is the span of the optimal bias function.

VII Experiments

We have performed our experiments on one dimensional and two dimensional state spaces. For one-dimensional case we implemented the algorithm on a simple problem with $A=2$ with $[0,5\pi]$ being the state space $S$. In this problem, the reward functions for the actions $a_{1},a_{2}$ are $\sin(x)$ and $-\sin(x)$ respectively. So the optimal policy is, for a state $x$ in interval $[n\pi,(n+1)\pi]$, the one which takes action $a_{2}$ if $n$ is odd and

$a_{1}$ if $n$ is even. The reward was scaled to the range $[0,1]$ to meet the requirements of [10]. From the current state $x$, after taking an action $a_{i}$ such that $i\in{1,2}$, the next state is sampled uniformly from the state space $S$.

We can see from Figure 6 that the empirical regret is converging faster for larger number of intervals, for the same horizon length The convergence can be better understood by comparing the the points where the empirical regret approaches to [math]. The very first point, in every plot, where the empirical regret is zero is indicated by a dashed vertical line showing the episode number. The results match the intuition that for the same problem having more number of states improves the performance.

By keeping $n$ constant and varying $H$ it can be seen from the plots that regret approaches zero slower as $H$ is increases. It can also be seen that the regret for larger horizon length at a particular episode was higher than the regret at the same episode for smaller horizon length. Thus the regret increases with the horizon length as expected.

Next we conducted experiments on a two-dimensional problem with $A=2$ and a Lipschitz continuous reward function for the respective actions. The reward functions were $\sin(\frac{x-y}{\sqrt{2}})+\cos(\frac{x+y}{\sqrt{2}})$ and $\sin(\frac{y-x}{\sqrt{2}})-\cos(\frac{x+y}{\sqrt{2}})$ for actions $a_{1}$ and $a_{2}$ respectively. The analysis of this case is also similar to the one-dimensional setting. The algorithm shows similar trend for different values of $H$ for same number of intervals $n$ (see figure 5) i.e, regret approaches zero slower for higher values of $H$ as in one-dimensional setting.

VIII Conclusion

We considered the finite-horizon continuous reinforcement learning problem. We have given an algorithm based on UCBVI for the same. With the only assumption that the reward function and transition probabilities are Lipscitz continuous we show that the upper bound on the discretization error is $const.Ln^{-\alpha}T$. We have shown the matching lower bound under the assumption that the algorithm discretizes the state-space of MDP. In future we would like to show similar bound without this assumption. Also, with an additional assumption that the sampling is unbiased, we proved that the regret is of order $O(T^{2/3})$, when using our algorithm.

We have also given some experimental results to validate our propositions. In future we would like to extend the algorithm to infinite-horizon case. This seems to be difficult as the UCBVI algorithm needs the horizon length $H$ as input. We also want to improve the dependence on size of action $A$, Lipschitz constant $L$ and horizon length $H$ on the regret.