Learning Policies for Markov Decision Processes from Data

TL;DR

This paper introduces a method for learning policies in Markov decision processes from data using regularized logistic regression, providing theoretical bounds on performance and demonstrating effectiveness on a synthetic navigation task.

Contribution

It develops a novel approach combining feature-based policy learning with theoretical guarantees on regret and sample complexity for MDPs.

Findings

Regret bounds relate to generalization error and Markov chain ergodicity.

Sample complexity scales logarithmically with feature count and polynomially with inverse accuracy.

Method successfully applied to a synthetic robot navigation example.

Abstract

We consider the problem of learning a policy for a Markov decision process consistent with data captured on the state-actions pairs followed by the policy. We assume that the policy belongs to a class of parameterized policies which are defined using features associated with the state-action pairs. The features are known a priori, however, only an unknown subset of them could be relevant. The policy parameters that correspond to an observed target policy are recovered using $\ell_1$-regularized logistic regression that best fits the observed state-action samples. We establish bounds on the difference between the average reward of the estimated and the original policy (regret) in terms of the generalization error and the ergodic coefficient of the underlying Markov chain. To that end, we combine sample complexity theory and sensitivity analysis of the stationary distribution of Markov chains. Our analysis suggests that to achieve regret within order $O(\sqrt{\epsilon})$, it suffices to use training sample size on the order of $\Omega(\log n \cdot poly(1/\epsilon))$, where $n$ is the number of the features. We demonstrate the effectiveness of our method on a synthetic robot navigation example.

Figures

Equations

$${\mu }_{\mathbf{\theta }}(a|x)=\frac{\exp \{{\mathbf{\theta }}^{\prime }\mathbf{\varphi }(x,a)\}}{{\sum }_{b\in \mathcal{A}}\exp \{{\mathbf{\theta }}^{\prime }\mathbf{\varphi }(x,b)\}},$$

$$\overline{R}(\mathbf{\theta })=\sum _{(a,x)}{\eta }_{\mathbf{\theta }}(x,a)R(x,a).$$

$$\mathrm{Reg}(\mathcal{S}({\mathbf{\theta }}^{\ast }))=\overline{R}({\mathbf{\theta }}^{\ast })-\overline{R}(\hat{\mathbf{\theta }}),$$

$$\begin{array}[]{rl}\max_{\boldsymbol{\theta}\in{\mathbb{R}}^{n}}&\sum_{i=1}^{m}\log\mu_{\boldsymbol{\theta}}(a_{i}|x_{i})\\ \text{s.t.}&\|\boldsymbol{\theta}\|_{1}\leq B,\end{array}$$

$$ϵ(\mathbf{\theta })={\mathbb{E}}_{(x,a)\sim \mathcal{D}}[-\log {\mu }_{\mathbf{\theta }}(a|x)],$$

$${\widehat{ϵ}}_{\mathcal{S}}(\mathbf{\theta })=\frac{1}{m}\sum _{i=1}^m(-\log {\mu }_{\mathbf{\theta }}(a_i|x_i)).$$

$$\widehat{ϵ}(\mathbf{\theta })\stackrel{△}{=}{\widehat{ϵ}}_{\mathcal{S}({\mathbf{\theta }}^{\ast })}(\mathbf{\theta }).$$

$$D({\mathbf{\mu }}_{{\mathbf{\theta }}_1}(\cdot |x)\Vert {\mathbf{\mu }}_{{\mathbf{\theta }}_2}(\cdot |x))=\sum _a{\mu }_{{\mathbf{\theta }}_1}(a|x)\log \frac{{\mu }_{{\mathbf{\theta }}_1}(a|x)}{{\mu }_{{\mathbf{\theta }}_2}(a|x)}.$$

$$D_{\mathbf{\theta }}({\mathbf{\mu }}_{{\mathbf{\theta }}_1}\Vert {\mathbf{\mu }}_{{\mathbf{\theta }}_2})=\sum _x{\pi }_{\mathbf{\theta }}(x)D({\mathbf{\mu }}_{{\mathbf{\theta }}_1}(\cdot |x)\Vert {\mathbf{\mu }}_{{\mathbf{\theta }}_2}(\cdot |x)).$$

$$ϵ(\hat{\mathbf{\theta }})-ϵ(\mathbf{\theta })=D_{\mathbf{\theta }}({\mathbf{\mu }}_{\mathbf{\theta }}\Vert {\mathbf{\mu }}_{\hat{\mathbf{\theta }}})$$

$$ϵ(\hat{\mathbf{\theta }})-ϵ(\mathbf{\theta })$$

$$D_{{\mathbf{\theta }}^{\ast }}({\mathbf{\mu }}_{{\mathbf{\theta }}^{\ast }}\Vert {\mathbf{\mu }}_{\hat{\mathbf{\theta }}})\le ϵ,$$

$$|ϵ(\hat{\mathbf{\theta }})-ϵ({\mathbf{\theta }}^{\ast })|<\epsilon ,$$

$$m=\Omega\Big{(}(\log n)\cdot\textrm{poly}(r,\,K,\,C,\,\,H,\,\log(1/\delta),\,1/\varepsilon)\Big{)}.$$

$${\mathcal{N}}_p(\mathcal{F},\epsilon ,m)=\underset{{\mathbf{z}}^{(1)},\dots ,{\mathbf{z}}^{(m)}}{\sup }{\mathcal{N}}_p(\mathcal{F},\epsilon ,({\mathbf{z}}^{(1)},\dots ,{\mathbf{z}}^{(m)})).$$

$$l(\mathbf{\varphi }(x),i)=-\log \left(\frac{\exp ({\mathbf{\theta }}^{\prime }{\mathbf{\varphi }}_i(x))}{{\sum }_{j=1}^H\exp ({\mathbf{\theta }}^{\prime }{\mathbf{\varphi }}_j(x))}\right).$$

$$\mathbb{P}\left[\exists f\in\mathcal{F}:\left|\frac{1}{m}\sum_{i=1}^{m}f({\mathbf{z}}^{(i)})-\mathbb{E}_{{\mathbf{z}}\sim D}[f({\mathbf{z}})]\right|\geq\varepsilon\right]\\ \leq 8\mathbb{E}\big{[}\mathcal{N}_{1}(\mathcal{F},\varepsilon/8,({\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}))\big{]}\exp\left(\frac{-m\varepsilon^{2}}{512M^{2}}\right).$$

$${\log }_2{\mathcal{N}}_2(\mathcal{G},\epsilon ,m)\le \frac{B^2{\zeta }^2}{{\epsilon }^2}{\log }_2(2n+1).$$

$$f\Big{(}\arg\min_{\boldsymbol{\theta}\in\Theta}\hat{f}(\boldsymbol{\theta})\Big{)}\leq\min_{\boldsymbol{\theta}\in\Theta}f(\boldsymbol{\theta})+2\varepsilon.$$

$$\mathcal{F}=\big{\{}f_{g}({\boldsymbol{\phi}}(x),y)=l(g({\boldsymbol{\phi}}_{1}(x)),\ldots,g({\boldsymbol{\phi}}_{H}(x)),y),g\in\mathcal{G},\ y\in\mathcal{A}\big{\}}.$$

$$\mathcal{N}_{1}(\mathcal{F},\varepsilon,m)\leq\big{[}\mathcal{N}_{1}(\mathcal{G},\varepsilon/(LH),m)\big{]}^{H}.$$

$${\mathbf{z}}^{(1)}=(\mathbf{\varphi }(x^{(1)}),y^{(1)}),\dots ,{\mathbf{z}}^{(m)}=(\mathbf{\varphi }(x^{(m)}),y^{(m)})$$

$${\mathcal{N}}_1(\mathcal{G},\epsilon /(LH),({\mathbf{\varphi }}_j(x^{(1)}),\dots ,{\mathbf{\varphi }}_j(x^{(m)})))\le \Gamma .$$

$$\Vert (g({\mathbf{\varphi }}_j(x^{(1)})),$$

$$=$$

$$\le$$

$$l(v_{1,1}^{(j_1)},v_{2,1}^{(j_2)},\dots ,v_{H,1}^{(j_H)},y^{(1)}),\dots ,l(v_{1,m}^{(j_1)},v_{2,m}^{(j_2)},\dots ,v_{H,m}^{(j_H)},y^{(m)}),$$

$$\displaystyle\bigg{\|}\big{(}f_{g}({\mathbf{z}}^{(1)}),\ldots,f_{g}({\mathbf{z}}^{(m)})\big{)}-$$

$$\displaystyle\hskip 28.45274pt\big{(}l(v_{1,1}^{k(1)},v_{2,1}^{k(2)},\ldots,v_{H,1}^{k(H)},y^{(1)}),\ldots,$$

$$\displaystyle\hskip 28.45274ptl(v_{1,m}^{k(1)},v_{2,m}^{k(2)},\ldots,v_{H,m}^{k(H)},y^{(m)})\big{)}\bigg{\|}_{1}$$

$$\displaystyle\leq L\bigg{\|}\bigg{(}\sum_{h=1}^{H}|v_{h,1}^{k(h)}-g({\boldsymbol{\phi}}_{h}(x^{(1)}))|,\ldots,$$

$$\displaystyle\hskip 28.45274pt\sum_{h=1}^{H}|v_{h,m}^{k(h)}-g({\boldsymbol{\phi}}_{h}(x^{(m)}))|\bigg{)}\bigg{\|}_{1}$$

$$=L\sum _{h=1}^H\sum _{i=1}^m|g({\mathbf{\varphi }}_h(x^{(i)}))-v_{h,i}^{k(h)}|\le m\epsilon ,$$

$$\mathcal{G}=\Big{\{}g:[0,1]^{n}\to{\mathbb{R}}\;\Big{|}\;g({\mathbf{x}})=\boldsymbol{\theta}^{\prime}{\mathbf{x}},\,\|\boldsymbol{\theta}\|_{1}\leq\hat{B}\Big{\}}.$$

Full text

1 Introduction

Markov Decision Processes (MDPs) offer a framework for many dynamic optimization problems under uncertainty [1, 2]. When the state-action space is not large and transition probabilities for all state-action pairs are known, standard techniques such as policy iteration and value iteration can compute an optimal policy. More often than not, however, problem instances of realistic size have very large state-action spaces and it becomes impractical to know the transition probabilities everywhere and compute an optimal policy using these off-line methods.

For such large problems, one resorts to approximate methods, collectively referred to as Approximate Dynamic Programming (ADP) [3, 4]. ADP methods approximate either the value function and/or the policy and optimize with respect to the parameters, as for instance is done in actor-critic methods [5, 6, 7]. The optimization of approximation parameters requires the learner to have access to the system and be able to observe the effect of applied control actions.

In this paper, we adopt a different perspective and assume that the learner has no direct access to the system but only has samples of the actions applied in various states of the system. These actions applied in various states are generated according to a policy that is fixed but unknown. As an example, the actions could be followed by an expert player who plays an optimal policy. Our goal is to learn a policy consistent with the observed states-actions, which we will call demonstrations.

Learning from an expert is a problem that has been studied in the literature and referred to as apprenticeship learning [8, 9], imitation learning [10] or learning from demonstrations [11]. While there are many settings where it could be useful, the main application driver has been robotics [12, 13, 14]. Additional interesting application examples include: learning from an experienced human pilot/driver to navigate vehicles autonomously, learning from animals to develop bio-inspired policies, and learning from expert players of a game to train a computer player. In all these examples, and given the size of the state-action space, we will not observe the actions of the expert in all states, or more broadly “scenarios” corresponding to parts of the state space leading to similar actions. Still, our goal is to learn a policy that generalizes well beyond the scenarios that have been observed and is able to select appropriate actions even at unobserved parts of the state space.

A plausible way to obtain such a policy is to learn a mapping of the states-actions to a lower dimensional space. Borrowing from ADP methods, we can obtain a lower-dimensional representation of the state-action space through the use of features that are functions of the state and the action taken at that state. In particular, we will consider policies that combine various features through a weight vector and reduce the problem of learning the policy to learning this weight/parameter vector. Essentially, we will be learning a parsimonious parametrization of the policy used by the expert.

The related work in the literature on learning through demonstrations can be broadly classified into direct and indirect methods [13]. In the direct methods, supervised learning techniques are used to obtain a best estimate of the expert’s behavior; specifically, a best fit to the expert’s behavior is obtained by minimizing an appropriately defined loss function. A key limitation of this method is that the estimated policy is not well adapted to the parts of the state space not visited often by the expert, thus resulting in poor action selection if the system enters these states. Furthermore, the loss function can be non-convex in the policy, rendering the corresponding problem hard to solve.

Indirect methods, on the other hand, evaluate a policy by learning the full description of the MDP. In particular, one solves a so called inverse reinforcement learning problem which assumes that the dynamics of the environment are known but the one-step reward function is unknown. Then, one estimates the reward function that the expert is aiming to maximize through the demonstrations, and obtains the policy simply by solving the MDP. A policy obtained in this fashion tends to generalize better to the states visited infrequently by the expert, compared to policies obtained by direct methods. The main drawback of inverse reinforcement learning is that at each iteration it requires to solve an MDP which is computationally expensive. In addition, the assumption that the dynamics of the environment are known for all states and actions is unrealistic for problems with very large state-action spaces.

In this work, we exploit the benefits of both direct and indirect methods by assuming that the expert is using a Randomized Stationary Policy (RSP). As we alluded to earlier, an RSP is characterized in terms of a vector of features associated with state-action pairs and a parameter $\boldsymbol{\theta}$ weighing the various elements of the feature vector. We consider the case where we have many features, but only relatively few of them are sufficient to approximate the target policy well. However, we do not know in advance which features are relevant; learning them and the associated weights (elements of $\boldsymbol{\theta}$) is part of the learning problem.

We will use supervised learning to obtain the best estimate of the expert’s policy. As in actor-critic methods, we use an RSP which is a parameterized “Boltzmann” policy and rely on an $\ell_{1}$-regularized maximum likelihood estimator of the policy parameter vector. An $\ell_{1}$-norm regularization induces sparse estimates and this is useful in obtaining an RSP which uses only the relevant features. In [15], it is shown that the sample complexity of $\ell_{1}$-penalized logistic regression grows as $\mathcal{O}(\log n)$, where $n$ is the number of features. As a result, we can learn the parameter vector $\boldsymbol{\theta}$ of the target RSP with relatively few samples, and the RSP we learn generalizes well across states that are not included in the demonstrations of the expert. Furthermore, $\ell_{1}$-regularized logistic regression is a convex optimization problem which can be solved efficiently.

1.1 Related Work

There is substantial work in the literature on learning MDP policies by observing experts; see [13] for a survey. We next discuss papers that are more closely related to our work.

In the context of indirect methods, [16] develops techniques for estimating a reward function from demonstrations under varying degrees of generality on the availability of an explicit model. [8] introduces an inverse reinforcement learning algorithm that obtains a policy from observed MDP trajectories followed by an expert. The policy is guaranteed to perform as well as that of the expert’s policy, even though the algorithm does not necessarily recover the expert’s reward function. In [9], the authors combine supervised learning and inverse reinforcement learning by fitting a policy to empirical estimates of the expert demonstrations over a space of policies that are optimal for a set of parameterized reward functions. [9] shows that the policy obtained generalizes well over the entire state space. [11] uses a supervised learning method to learn an optimal policy by leveraging the structure of the MDP, utilizing a kernel-based approach. Finally, [10] develops the DAGGER (Dataset Aggregation) algorithm that trains a deterministic policy which achieves good performance guarantees under its induced distribution of states.

Our work is different in that we focus on a parameterized set of policies rather than parameterized rewards. In this regard, our work is similar to approximate DP methods which parameterize the policy, e.g., expressing the policy as a linear functional in a lower dimensional parameter space. This lower-dimensional representation is critical in overcoming the well known “curse of dimensionality.”

1.2 Contributions

We adopt the $\ell_{1}$-regularized logistic regression to estimate a target RSP that generates a given collection of state-action samples. We evaluate the performance of the estimated policy and derive a bound on the difference between the average reward of the estimated RSP and the target RSP, typically referred to as regret. We show that a good estimation of the parameter of the target RSP also implies a good bound on the regret. To that end, we generalize a sample complexity result on the log-loss of the maximum likelihood estimates [15] from the case of two actions available at each state to the multi-action case. Using this result, we establish a sample complexity result on the regret.

Our analysis is based on the novel idea of separating the loss in average reward into two parts. The first part is due to the error in the policy estimation (training error) and the second part is due to the perturbation in the stationary distribution of the Markov chain caused by using the estimated policy instead of the true target policy (perturbation error). We bound the first part by relating the result on the log-loss error to the Kulback-Leibler divergence [17] between the estimated and the target RSPs. The second part is bounded using the ergodic coefficient of the induced Markov chain. Finally, we evaluate the performance of our method on a synthetic example.

The paper is organized as follows. In Section 2, we introduce some of our notation and state the problem. In Section 3, we describe the supervised learning algorithm used to train the policy. In Section 4, we establish a result on the (log-loss) error in policy estimation. In Section 5, we establish our main result which is a bound on the regret of the estimated policy. In Section 7, we introduce a robot navigation example and present our numerical results. We end with concluding remarks in Section 8.

Notational conventions. Bold letters are used to denote vectors and matrices; typically vectors are lower case and matrices upper case. Vectors are column vectors, unless explicitly stated otherwise. Prime denotes transpose. For the column vector ${\mathbf{x}}\in\mathbb{R}^{n}$ we write ${\mathbf{x}}=(x_{1},\ldots,x_{n})$ for economy of space, while $\|{\mathbf{x}}\|_{p}=(\sum_{i=1}^{n}|x_{i}|^{p})^{1/p}$ denotes its $p$-norm. Vectors or matrices with all zeroes are written as ${\mathbf{0}}$, the identity matrix as ${\mathbf{I}}$, and $\mathbf{e}$ is the vector with all entries set to $1$. For any set $\mathcal{S}$, $|\mathcal{S}|$ denotes its cardinality. We use $\log$ to denote the natural logarithm and a subscript to denote different bases, e.g., $\log_{2}$ denotes logarithm with base $2$.

2 Problem Formulation

Let $(\mathcal{X},\mathcal{A},\boldsymbol{P},R)$ denote a Markov Decision Process (MDP) with a finite set of states $\mathcal{X}$ and a finite set of actions $\mathcal{A}$. For a state-action pair $(x,a)\in\mathcal{X}\times\mathcal{A}$, let $P(y|x,a)$ denote the probability of transitioning to state $y\in\mathcal{X}$ after taking action $a$ in state $x$. The function $R:\mathcal{X}\times\mathcal{A}\rightarrow\mathbb{R}$ denotes the one-step reward function.

Let us now define a class of Randomized Stationary Policies (RSPs) parameterized by vectors $\boldsymbol{\theta}\in\mathbb{R}^{n}$. Let $\{\mu_{\boldsymbol{\theta}}:\boldsymbol{\theta}\in\mathbb{R}^{n}\}$ denote the set of RSPs we are considering. For a given parameter $\boldsymbol{\theta}$ and a state $x\in\mathcal{X}$, $\mu_{\boldsymbol{\theta}}(a|x)$ denotes the probability of taking action $a$ at state $x$. Specifically, we consider the Boltzmann-type RSPs of the form

[TABLE]

where ${\boldsymbol{\phi}}:\mathcal{X}\times\mathcal{A}\rightarrow[0,\,1]^{n}$ is a vector of features associated with each state-action pair $(x,a)$. (Features are normalized to take values in $[0,1]$.) Henceforth, we identify an RSP by its parameter $\boldsymbol{\theta}$. We assume that the policy is sparse, that is, the vector $\boldsymbol{\theta}$ has only $r<n$ non-zero components and each is bounded by $K$, i.e., $|\theta_{i}|<K$ for all $i$. Given an RSP $\boldsymbol{\theta}$, the resulting transition probability matrix on the Markov chain is denoted by $\boldsymbol{P}_{\boldsymbol{\theta}}$, whose $(x,y)$ element is $P_{\boldsymbol{\theta}}(y|x)=\sum_{a\in\mathcal{A}}\mu_{\boldsymbol{\theta}}(a|x)P(y|x,a)$ for all state pairs $(x,y)$.

Notice that for any RSP $\boldsymbol{\theta}$, the sequence of states $\{X_{k}\}$ and the sequence of state-action pairs $\{X_{k},A_{k}\}$ form a Markov chain with state space $\mathcal{X}$ and $\mathcal{X}\times\mathcal{A}$, respectively. We assume that for every $\boldsymbol{\theta}$, the Markov chains $\{X_{k}\}$ and $\{X_{k},A_{k}\}$ are irreducible and aperiodic with stationary probabilities $\pi_{\boldsymbol{\theta}}(x)$ and $\eta_{\boldsymbol{\theta}}(x,a)=\pi_{\boldsymbol{\theta}}(x)\mu_{\boldsymbol{\theta}}(a|x)$, respectively.

The average reward function associated with an RSP $\boldsymbol{\theta}$ is a function $\overline{R}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ defined as

[TABLE]

Let now fix a target RSP $\boldsymbol{\theta}^{*}$. As we assumed above, $\boldsymbol{\theta}^{*}$ is sparse having at most $r$ non-zero components $\theta^{*}_{i}$, each satisfying $|\theta^{*}_{i}|\leq K$. This is simply the policy used by an expert (not necessarily optimal) which we wish to learn. We let $\mathcal{S}(\boldsymbol{\theta}^{*})=\{(x_{i},a_{i}):i=1,2,\ldots,m\}$ denote a set of state-action samples generated by playing policy $\boldsymbol{\theta}^{*}$. The state samples $\{x_{i}:i=1,2,\ldots,m\}$ are independent and identically distributed (i.i.d.) drawn from the stationary distribution ${\boldsymbol{\pi}}_{\boldsymbol{\theta}^{*}}$ and $a_{i}$ is the action taken in state $x_{i}$ according to the policy $\mu_{\boldsymbol{\theta}^{*}}$. It follows that the samples in $\mathcal{S}(\boldsymbol{\theta}^{*})$ are i.i.d. according to the distribution $\mathcal{D}\sim\eta_{\boldsymbol{\theta}^{*}}(x,a)$.

We assume we have access only to the demonstrations $\mathcal{S}(\boldsymbol{\theta}^{*})$ while the transition probability matrix $\boldsymbol{P}_{\boldsymbol{\theta}^{*}}$ and the target RSP $\boldsymbol{\theta}^{*}$ are unknown. The goal is to learn the target policy $\boldsymbol{\theta}^{*}$ and characterize the average reward obtained when the learned RSP is applied to the Markov process. In particular, we are interested in estimating the target parameter $\boldsymbol{\theta}^{*}$ from the samples efficiently and evaluate the performance of the estimated RSP with respect to the target RSP, i.e., bound the regret defined as

[TABLE]

where $\hat{\boldsymbol{\theta}}$ is the estimated RSP from the samples $\mathcal{S}(\boldsymbol{\theta}^{*})$.

3 Estimating the policy

Next we discuss how to estimate the target RSP $\boldsymbol{\theta}^{*}$ from the $m$ i.i.d. state-action training samples in $\mathcal{S}(\boldsymbol{\theta}^{*})$. Given the Boltzmann structure of the RSP we have assumed, we fit a logistic regression function using a regularized maximum likelihood estimator as follows:

[TABLE]

where $B$ is a parameter that adjusts the trade-off between fitting the training data “well” and obtaining a sparse RSP that generalizes well on a test sample.

We can evaluate how well the maximum likelihood function fits the samples in the logistic function using a log-loss metric, defined as the expected negative of the likelihood over the (random) test data. Formally, for any parameter $\boldsymbol{\theta}$, log-loss is given by

[TABLE]

where the expectation is taken over state-action pairs $(x,y)$ drawn from the distribution $\mathcal{D}$; recall that we defined $\mathcal{D}$ to be the stationary distribution $\eta_{\boldsymbol{\theta}^{*}}(x,a)$ of the state-action pairs induced by the policy $\boldsymbol{\theta}^{*}$. Since the expectation is taken with respect to new data not included in the training set $\mathcal{S}(\boldsymbol{\theta}^{*})$, we can think of $\epsilon(\boldsymbol{\theta})$ as an out-of-sample metric of how well the RSP $\boldsymbol{\theta}$ approximates the actions taken by the target RSP $\boldsymbol{\theta}^{*}$.

We also define a sample-average version of log-loss: given any set $\mathcal{S}=\{(x_{i},a_{i}):i=1,2,\ldots,m\}$ of state-action pairs, define

[TABLE]

We will use the term empirical log-loss to refer to log-loss over the training set $\mathcal{S}(\boldsymbol{\theta}^{*})$ and use the notation

[TABLE]

To estimate an RSP, say $\hat{\boldsymbol{\theta}}$, from the training set, we adopt the logistic regression with $\ell_{1}$ regularization introduced in [15]. The specific steps are shown in Algorithm 1.

4 Log-loss performance

In this section we establish a sample complexity result indicating that relatively few training samples (logarithmic in the dimension $n$ of the RSP parameter $\boldsymbol{\theta}$) are needed to guarantee that the estimated RSP $\hat{\boldsymbol{\theta}}$ has out-of-sample log-loss close to the target RSP $\boldsymbol{\theta}^{*}$. We will use the line of analysis in [15] but generalize the result from the case where only two actions are available for every state to the general multi-action setting.

We start by relating the difference of the log-loss function associated with RSP $\boldsymbol{\theta}$ and its estimate $\hat{\boldsymbol{\theta}}$ to the relative entropy, or Kullback-Leibler (KL) divergence, between the corresponding RSPs. For a given $x\in\mathcal{X}$, we denote the KL-divergence between RSPs, $\boldsymbol{\theta}_{1}$ and $\boldsymbol{\theta}_{2}$ as $D(\boldsymbol{\mu}_{\boldsymbol{\theta}_{1}}(\cdot|x)\|\boldsymbol{\mu}_{\boldsymbol{\theta}_{2}}(\cdot|x))$, where $\boldsymbol{\mu}_{\boldsymbol{\theta}}(\cdot|x)$ denotes the probability distribution induced by RSP $\boldsymbol{\theta}$ on $\mathcal{A}$ in state $x$, and is given as follows:

[TABLE]

We also define the average KL-divergence, denoted by $D_{\boldsymbol{\theta}}(\boldsymbol{\mu}_{\boldsymbol{\theta}_{1}}\|\boldsymbol{\mu}_{\boldsymbol{\theta}_{2}})$, as the average of $D(\boldsymbol{\mu}_{\boldsymbol{\theta}_{1}}(\cdot|x)\|\boldsymbol{\mu}_{\boldsymbol{\theta}_{2}}(\cdot|x))$ over states visited according to the stationary distribution $\boldsymbol{\pi}_{\boldsymbol{\theta}}$ of the Markov chain induced by policy $\boldsymbol{\theta}$. Specifically, we define

[TABLE]

Lemma 4.1

Let $\hat{\boldsymbol{\theta}}$ be an estimate of RSP $\boldsymbol{\theta}$. Then,

[TABLE]

Proof:

[TABLE]

∎

For the case of a binary action at every state, [15] showed the following result. To state the theorem, let $poly(\cdot)$ denote a function which is polynomial in its arguments and recall that $m$ is the number of state-action pairs in the training set $\mathcal{S}(\boldsymbol{\theta}^{*})$ used to learn $\hat{\boldsymbol{\theta}}$.

Theorem 4.2** **([15], Thm. 3.1)

Suppose action set $\mathcal{A}$ contains only two actions, i.e. $\mathcal{A}=\{0,1\}$. Let $\epsilon>0$ and $\delta>0$. In order to guarantee that, with probability at least $1-\delta$, $\hat{\boldsymbol{\theta}}$ produced by the algorithm performs as well as $\boldsymbol{\theta}^{*}$, i.e.,

[TABLE]

it suffices that $m=\Omega((\log n)\cdot poly(r,K,C,\log(1/\delta),1/\epsilon))$.

We will generalize the result to the case when more than two actions are available at each state. We assume that $|\mathcal{A}|=H$, that is, at most $H$ actions are available at each state. By introducing in (1) features that get activated at specific states, it is possible to accommodate MDPs where some of the actions are not available at these states.

Theorem 4.3

Let $\varepsilon>0$ and $\delta>0$. In order to guarantee that, with probability at least $1-\delta$, $\hat{\boldsymbol{\theta}}$ produced by Algorithm 1 performs as well as $\boldsymbol{\theta}^{*}$, i.e.,

[TABLE]

it suffices that

[TABLE]

Furthermore, in terms of only $H$, $m=\Omega(H^{3})$.

The remainder of this section is devoted to proving Theorem 4.3. The proof is similar to the proof of Theorem 4.2 (Theorem 3.1 in [15]) but with key differences to accommodate the multiple actions per state. We start by introducing some notations and stating necessary lemmata.

Denote by $\mathcal{F}$ a class of functions over some domain $\mathcal{D}_{\mathcal{F}}$ and range $[-M,M]\subset{\mathbb{R}}$. Let $\mathcal{F}|_{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}}=\{(f({\mathbf{z}}^{(1)}),\ldots,f({\mathbf{z}}^{(m)}))\mid f\in\mathcal{F}\}\subset[-M,M]^{m}$, which is the valuation of the class of functions at a certain collection of points ${\mathbf{z}}^{(1)},\ldots,\linebreak[3]{\mathbf{z}}^{(m)}\in\mathcal{D}_{\mathcal{F}}$. It is said that a set of vectors $\{{\mathbf{v}}^{(1)},\ldots,{\mathbf{v}}^{(k)}\}$ in ${\mathbb{R}}^{m}$ $\varepsilon$-covers $\mathcal{F}|_{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}}$ in the $p$-norm, if for every point ${\mathbf{v}}\in\mathcal{F}|_{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}}$ there exists some ${\mathbf{v}}^{(i)}$, $i=1,\ldots,k$, such that $\|{\mathbf{v}}-{\mathbf{v}}^{(i)}\|_{p}\leq m^{1/p}\varepsilon$. Let also denote $\mathcal{N}_{p}(\mathcal{F},\varepsilon,({\mathbf{z}}^{(1)},\,\ldots,\,{\mathbf{z}}^{(m)}))$ the size of the smallest set that $\varepsilon$-covers $\mathcal{F}|_{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}}$ in the $p$-norm. Finally, let

[TABLE]

To simplify the representation of log-loss of the general logistic function in (1), we use the following notations. For each $x\in\mathcal{X}$, ${\boldsymbol{\phi}}(x)=({\boldsymbol{\phi}}_{1}(x),\ldots,{\boldsymbol{\phi}}_{H}(x))\linebreak[3]=({\boldsymbol{\phi}}(x,1),\ldots,{\boldsymbol{\phi}}(x,H))\in[0,1]^{nH}$ denotes feature vectors associated with each action. For any $\boldsymbol{\theta}\in{\mathbb{R}}^{n}$, define the $\log$-likelihood function $l:[0,1]^{nH}\times\{1,\ldots,H\}\to{\mathbb{R}}$ as

[TABLE]

Note $\mu_{\boldsymbol{\theta}}(a|x)=\exp\{-l({\boldsymbol{\phi}}(x),a)\}$. Further, let $g:[0,1]^{n}\rightarrow{\mathbb{R}}$ be the class of functions $g({\mathbf{x}})=\boldsymbol{\theta}^{\prime}{\mathbf{x}}$. We can then rewrite $l$ using $g$ as $l({\boldsymbol{\phi}}(x),y)=l(g({\boldsymbol{\phi}}_{1}(x)),\linebreak[3]\ldots,g({\boldsymbol{\phi}}_{H}(x)),y)$.

Lemma 4.4** **([15, 18])

Let there be some distribution $D$ over $\mathcal{D}_{\mathcal{F}}$ and suppose ${\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}$ are drawn from $D$ i.i.d. Then,

[TABLE]

Lemma 4.5** **([19])

Suppose $\mathcal{G}=\{g:g({\mathbf{x}})=\boldsymbol{\theta}^{\prime}{\mathbf{x}},\ {\mathbf{x}}\in{\mathbb{R}}^{n},\ \|\boldsymbol{\theta}\|_{q}\leq B\}$ and the input ${\mathbf{x}}\in{\mathbb{R}}^{n}$ has a norm-bound such that $\|{\mathbf{x}}\|_{p}\leq\zeta$, where $1/p+1/q=1$. Then

[TABLE]

Lemma 4.6** **([15, 20])

If $|f(\boldsymbol{\theta})-\hat{f}(\boldsymbol{\theta})|\leq\varepsilon$ for all $\boldsymbol{\theta}\in\Theta$, then

[TABLE]

Lemma 4.7

Let $\mathcal{G}$ be a class of functions from ${\mathbb{R}}^{n}$ to some bounded set in ${\mathbb{R}}$. Consider $\mathcal{F}$ as a class of functions from ${\mathbb{R}}^{H}\times\mathcal{A}$ to some bounded set in ${\mathbb{R}}$, with the following form:

[TABLE]

If $l(\cdot,y)$ is Lipschitz with Lipschitz constant $L$ in the $\ell_{1}$-norm for every $y\in\mathcal{A}$, then we have

[TABLE]

**Proof: **Let $\Gamma=\mathcal{N}_{1}(\mathcal{G},\varepsilon/(LH),m)$. It is sufficient to show for every $m$ inputs

[TABLE]

we can find $\Gamma^{H}$ points in ${\mathbb{R}}^{m}$ that $\varepsilon$-cover $\mathcal{F}|_{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}}$. Fix some set of points ${\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}\subset{\mathbb{R}}^{nH+1}$. From the definition of $\mathcal{N}_{1}(\mathcal{G},\varepsilon/(LH),m)$, for each $j\in\{1,\ldots,H\}$,

[TABLE]

Let 111One may find less than $\Gamma$ points, but we consider the worst case scenario. $\{{\mathbf{v}}_{j}^{(1)},\ldots,{\mathbf{v}}_{j}^{(\Gamma)}\}$ be a set of $\Gamma$ points in ${\mathbb{R}}^{m}$ that $\varepsilon/(LH)$-covers $\mathcal{G}|_{{\boldsymbol{\phi}}_{j}(x^{(1)}),\ldots,{\boldsymbol{\phi}}_{j}(x^{(m)})}$. We use notation $v_{j,i}^{k}$ to denote the $i$th element of vector ${\mathbf{v}}_{j}^{k}$. Then, for any $g\in\mathcal{G}$ and $j\in\{1,\ldots,H\}$, there exists a $k(j)\in\{1,\ldots,\Gamma\}$ such that

[TABLE]

Now consider $\Gamma^{H}$ points with the following form

[TABLE]

where $j_{1},\ldots,j_{H}\in\{1,\ldots,\Gamma\}$.

Given a $g\in\mathcal{G}$ and $f_{g}(\cdot)\in\mathcal{F}$, let $k(1),\ldots k(H)\in\{1,\ldots,\Gamma\}$ be as defined above and consider

[TABLE]

where we used the Lipschitz property of the $l(\cdot)$ function in the first inequality and the last inequality follows from (4). Thus, the $\Gamma^{H}$ points $\epsilon$-cover $\mathcal{F}|_{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}}$ in $\ell_{1}$-norm. Finally, notice that the set of $m$ points $\{{\mathbf{z}}^{(1)},\ldots,{\mathbf{z}}^{(m)}\}$ is arbitrary, which concludes the proof. ∎

We now continue with the proof of Theorem 4.3. Recall Algorithm 1 and let $\hat{B}$ be the smallest integer in $\{0,1,2,4,\ldots\}$ that is greater or equal to $rK$. Notice that in Algorithm 1 one can use a larger $C$ but we select the smallest possible to obtain a tighter bound. For such a $\hat{B}$, it follows that $rK\leq\hat{B}\leq 2rK$. Define a class of functions $\mathcal{G}$ with domain $[0,1]^{n}$ as

[TABLE]

By Lemma 4.5 and Eq. (8),

[TABLE]

The partial derivatives of the log-loss function are

[TABLE]

and it can be seen that

[TABLE]

Hence, the Lipschitz constant for $l(\cdot,y)$ is $1$ for any $y=1,\ldots,H$.

By Lemma 4.7, we have

[TABLE]

Using the relation $\mathcal{N}_{1}\leq\mathcal{N}_{2}$ ([15, 19, 20]), we obtain

[TABLE]

We next find the range of class $\mathcal{F}$. To begin with, the range of class $\mathcal{G}$ is

[TABLE]

Since $l(\cdot,i)$ is Lipschitz in $\ell_{1}$-norm with Lipschitz constant $1$ and $|f(0,\ldots,0,y)|\linebreak[3]=\log(H)<H$ (by the fact $H\geq 2$), then

[TABLE]

which implies

[TABLE]

Finally, let $m_{1}=(1-\gamma)m$, which is the size of training set in Algorithm 1. From Lemma 4.4, Eq. (12) and Eq. (11), we have

[TABLE]

Treat $(1-\gamma)$ as a constant. To upper bound the right hand side of the above equation by $\delta$, it suffices to have

[TABLE]

The rest of the proof follows closely the proof of Theorem 3.1 in [15]. We outline the key steps for the sake of completeness. Suppose $m$ satisfies (14); then, with probability at least $1-\delta$, for all $f\in\mathcal{F}$

[TABLE]

Thus, using our definition of $\mathcal{F}$ in (9), for any $\boldsymbol{\theta}$ with $\|\boldsymbol{\theta}\|_{1}\leq\hat{B}$ and with probability at least $1-\delta$, we have

[TABLE]

Therefore, for all $\boldsymbol{\theta}$ with $\|\boldsymbol{\theta}\|_{1}\leq\hat{B}$ and with probability at least $1-\delta$, it holds

[TABLE]

where $\mathcal{S}_{1}$ is the training set from Algorithm 1.

Essentially, we have shown that for $m$ large enough the empirical log-loss function $\hat{\epsilon}_{\mathcal{S}_{1}}(\cdot)$ is a good estimate of the log-loss function $\epsilon(\cdot)$. According to Step 2 of Algorithm 1, $\hat{\boldsymbol{\theta}}=\arg\min_{\boldsymbol{\theta}:\|\boldsymbol{\theta}\|_{1}\leq\hat{B}}\hat{\epsilon}_{\mathcal{S}_{1}}(\theta)$. By Lemma 4.6, we have

[TABLE]

where $\boldsymbol{\theta}^{*}$ is the target policy and the last inequality follows simply from the fact that $\|\boldsymbol{\theta}^{*}\|_{1}\leq rK\leq\hat{B}$.

Eq. 15 indicates that the training step of Algorithm 1, finds at least one parameter vector $\hat{\boldsymbol{\theta}}$ whose performance is nearly as good as that of $\boldsymbol{\theta}^{*}$. At the validation step, we select one of the $\boldsymbol{\theta}_{B}$ found during training. It can be shown ([20]) that with a validation set of the same order of magnitude as the training set (and independent of $n$), we can ensure that with probability at least $1-\delta$, the selected parameter vector will have performance at most $2\varepsilon$ worse than that of the best performing vector discovered in the training step. Hence, with probability at least $1-2\delta$, the output $\hat{\boldsymbol{\theta}}$ of our algorithm satisfies

[TABLE]

Finally, replacing $\delta$ with $\delta/2$ and $\varepsilon$ with $\varepsilon/4$ everywhere in the proof, establishes Theorem 4.3.

5 Bounds on Regret

Theorem 4.3 provides a sufficient condition on the number of samples required to learn a policy whose log-loss performance is close to the target policy. In this section we study the regret of the estimated policy, defined in Sec. 2 as the difference between the average reward of the target policy and the estimated policy. Given that we use a number of samples in the training set proportional to the expression provided in Theorem 4.3, we establish explicit bounds on the regret.

We will bound the regret of the estimated policy by separating the effect of the error in estimating the policy function (which is characterized by Theorem 4.3) and the effect the estimated policy function introduces in the stationary distribution of the Markov chain governing how states are visited. To bound the regret due to the perturbation of the stationary distribution, we will use results from the sensitivity analysis of Markov chains. In Section 5.1 we provide some standard definitions for Markov chains and state our result on the regret, while in Section 6 we provide a proof of this result.

5.1 Main Result

We start by defining the fundamental matrix of a Markov chain.

Definition 1

The fundamental matrix of a Markov chain with state transition probability matrix $\boldsymbol{P}_{\boldsymbol{\theta}}$ induced by RSP $\boldsymbol{\theta}$ is

[TABLE]

where $\boldsymbol{e}$ denotes the vector of all $1$’s, $\boldsymbol{A}_{\boldsymbol{\boldsymbol{\theta}}}=\boldsymbol{I}-\boldsymbol{P}_{\boldsymbol{\boldsymbol{\theta}}}$ and $\boldsymbol{\pi}_{\boldsymbol{\boldsymbol{\theta}}}$ denotes the stationary distribution associated with $\boldsymbol{P}_{\boldsymbol{\theta}}$. Also, the group inverse of $\boldsymbol{A}_{\boldsymbol{\boldsymbol{\theta}}}$ denoted as ${\mathbf{A}}^{\#}_{\boldsymbol{\boldsymbol{\theta}}}$ is the unique matrix satisfying

[TABLE]

Most of the properties of a Markov chain can be expressed in terms of the fundamental matrix $\boldsymbol{Z}$. For example, if ${\boldsymbol{\pi}}_{1}$ is the stationary probability distribution associated with a Markov chain whose transition probability matrix is $\boldsymbol{P}_{1}$ and if $\boldsymbol{P}_{2}=\boldsymbol{P}_{1}+\boldsymbol{E}$ for some perturbation matrix $\boldsymbol{E}$, then the stationary probability distribution ${\boldsymbol{\pi}}_{2}$ of the Markov chain with transition probability matrix $\boldsymbol{P}_{2}$ satisfies the relation

[TABLE]

where $\boldsymbol{Z}_{1}$ is the fundamental matrix associated with $\boldsymbol{P}_{1}$.

Definition 2

The ergodic coefficient of a matrix $\boldsymbol{B}$ with equal row sums is

[TABLE]

The ergodic coefficient of a Markov chain indicates sensitivity of its stationary distribution. For any stochastic matrix $\boldsymbol{P}$, $0\leq\tau(\boldsymbol{P})\leq 1$.

We now have all the ingredients to state our main result bounding regret.

Theorem 5.1

Given $\epsilon>0$ and $\delta>0$, suppose

[TABLE]

i.i.d. samples are used by Algorithm 1 to produce an estimate $\hat{\boldsymbol{\theta}}$ the unknown target RSP policy parameter $\boldsymbol{\theta}^{*}$. Then with probability at least $1-\delta$, we have

[TABLE]

where

[TABLE]

and $\kappa$ is a constant that depends on the RSP $\hat{\boldsymbol{\theta}}$ and can be any of the following:

• •

$\kappa=\|\boldsymbol{Z}_{\hat{\boldsymbol{\boldsymbol{\theta}}}}\|_{1}$,

• •

$\kappa=\|\boldsymbol{A}^{\#}_{\hat{\boldsymbol{\boldsymbol{\theta}}}}\|_{1}$,

• •

$\kappa=1/(1-\tau(\boldsymbol{P}_{\hat{\boldsymbol{\boldsymbol{\theta}}}}))$,

• •

$\kappa=\tau(\boldsymbol{Z}_{\hat{\boldsymbol{\boldsymbol{\theta}}}})=\tau(\boldsymbol{A}^{\#}_{\hat{\boldsymbol{\boldsymbol{\theta}}}})$.

The constant $\kappa$ is referred to as condition number. The regret is thus governed by the condition number of the estimated RSP; the smaller the condition number of the trained policy, the smaller is the regret.

6 Proof of the Main Result

In this section we analyze the average reward obtained by the MDP when we apply the estimated RSP $\hat{\boldsymbol{\theta}}$, and prove the regret bound in Theorem 5.1. First, we bound the regret as the sum of two parts.

[TABLE]

Note that the first absolute sum above has terms $\sum_{a}[\mu_{\boldsymbol{\theta}^{*}}(a|x)-\mu_{\hat{\boldsymbol{\theta}}}(a|x)]$ for all $x$ that are related to the estimation error from fitting the RSP policy $\hat{\boldsymbol{\theta}}$ to $\boldsymbol{\theta}^{*}$. The second part has terms $\sum_{x}|\pi_{\hat{\boldsymbol{\theta}}}(x)-\pi_{\boldsymbol{\theta}^{*}}(x)|$ that are related to the perturbation of the stationary distribution of the Markov chain by applying the fitted RSP $\hat{\boldsymbol{\theta}}$ instead of the original $\boldsymbol{\theta}^{*}$. In the following, we bound each term separately. We begin with the first term.

[TABLE]

where $\boldsymbol{\mu}_{\boldsymbol{\theta}}(\cdot|x)$ denotes the probability distribution (a vector) on the action space $\mathcal{A}$ induced by the RSP $\boldsymbol{\theta}$ at state $x$.

The bound in (20) is related to the difference in the log-loss between the RSPs $\boldsymbol{\theta}^{*}$ and $\hat{\boldsymbol{\theta}}$. To see this, we need the following result that connects the $\ell_{1}$ distance between two distributions with their KL-divergence. Let $\boldsymbol{p}_{1}$ and $\boldsymbol{p}_{2}$ denote two probability vectors on $\mathcal{A}$. From the variation distance characterization of $\boldsymbol{p}_{1}$ and $\boldsymbol{p}_{2}$ we have the following lemma.

Lemma 6.1

([17, Lemma 11.6.1])

[TABLE]

Continuing the chain of inequalities from (20), we obtain

[TABLE]

In the first inequality, we applied Lemma 6.1 by setting $\boldsymbol{p}_{1}=\boldsymbol{\mu}_{\boldsymbol{\theta}^{*}}(\cdot|x)$ and $\boldsymbol{p}_{2}=\boldsymbol{\mu}_{\hat{\boldsymbol{\theta}}}(\cdot|x)$ for each $x$. In second inequality, we applied Jensen’s inequality. We can now use Theorem 4.3 to bound (22).

We next bound the second term in (19) using techniques from perturbation analysis of eigenvalues of a matrix. We have

[TABLE]

where we used the definition of $R_{\max}$ in the 2nd inequality and the last equality follows by noting that $\sum_{a}\mu_{\hat{\boldsymbol{\theta}}}(a|x)=1$ for all $x$. To further bound the difference in stationary distributions in (23), we use a relation between a measure of perturbation of the stationary distribution and the condition number of Markov chains. We recall the following result that is useful to bound the difference between the stationary distribution induced by the optimal RSP and that induced by the estimated RSP in terms of the condition number of the Markov chains.

Lemma 6.2** **([21, 22])

Let $\boldsymbol{\pi}_{\boldsymbol{\boldsymbol{\theta}}_{1}}$ and $\boldsymbol{\pi}_{\boldsymbol{\boldsymbol{\theta}}_{2}}$ be the unique stationary distributions of the stochastic matrices $\boldsymbol{P}_{\boldsymbol{\boldsymbol{\theta}}_{1}}$ and $\boldsymbol{P}_{\boldsymbol{\boldsymbol{\theta}}_{2}}$, respectively. Let $\boldsymbol{E}=\boldsymbol{P}_{\boldsymbol{\boldsymbol{\theta}}_{1}}-\boldsymbol{P}_{\boldsymbol{\boldsymbol{\theta}}_{2}}$. Then,

[TABLE]

where $\kappa$ is a constant that can take the following values

• •

$\kappa=\|\boldsymbol{Z}_{\boldsymbol{\boldsymbol{\theta}}_{2}}\|_{1}$,

• •

$\kappa=\|\boldsymbol{A}^{\#}_{\boldsymbol{\boldsymbol{\theta}}_{2}}\|_{1}$,

• •

$\kappa=1/(1-\tau(\boldsymbol{P}_{\boldsymbol{\boldsymbol{\theta}}_{2}}))$,

• •

$\kappa=\tau(\boldsymbol{Z}_{\boldsymbol{\boldsymbol{\theta}}_{2}})=\tau(\boldsymbol{A}^{\#}_{\boldsymbol{\boldsymbol{\theta}}_{2}})$.

**Proof: **The proof follows by setting $\boldsymbol{\pi}_{1}=\boldsymbol{\pi}_{\boldsymbol{\theta}_{2}}$ and $\boldsymbol{\pi}_{2}=\boldsymbol{\pi}_{\boldsymbol{\theta}_{1}}$ in (17) and using the relation

[TABLE]

The other relations follow similarly from Sec. 3 of [22]. ∎

Continuing the chain of inequalities in (23) and applying Lemma 6.2 by setting $\pi_{\boldsymbol{\boldsymbol{\theta}}_{1}}=\pi_{\boldsymbol{\theta}^{*}}$ and $\pi_{\boldsymbol{\boldsymbol{\theta}}_{2}}=\pi_{\hat{\boldsymbol{\theta}}}$, we obtain

[TABLE]

where, with some overloading of notation, $\kappa$ is now as specified in the expressions provided in the statement of Theorem 5.1.

The $i$th component of the vector $\boldsymbol{\pi}_{\boldsymbol{\theta}^{*}}^{\prime}(\boldsymbol{P}_{\boldsymbol{\theta}^{*}}-\boldsymbol{P}_{\hat{\boldsymbol{\theta}}})$ is given by

[TABLE]

where, in the last equality, we applied the definition of the transition probability $\boldsymbol{P}_{\boldsymbol{\theta}}$ associated with RSP $\boldsymbol{\theta}$.

It follows

[TABLE]

where the last inequality follows by noting that $\sum_{y}\boldsymbol{P}(y|x,a)=1$ for all $(x,a)$.

Now, using (26), (23), similar steps as in (22), and Theorem 4.3, we can bound the second term in (19) as

[TABLE]

Finally, combining (22) and (27) and applying Theorem 4.3, the result in Theorem 5.1 follows.

7 A robot navigation example

In this section, we discuss the experimental setup to simulate an MDP and validate the effectiveness of our proposed learning algorithm. We consider the problem of learning the policy used by an agent (a robot) as it moves on a 2-dimensional grid. After simulating the movement of the robot and recording its actions in various states, we use these states-action samples to learn the policy of the robot and evaluate its performance.

7.1 Environment and Agent Settings

Consider an agent moving in a $13\times 13$ grid, shown in Fig. 1. The agent’s position is specified by a two-tuple state ${\mathbf{x}}=(x_{1},x_{2})$, representing its coordinates in the grid. We assume $(0,0)$ is at the southwest corner of the grid and we make the convention that coordinates ${\mathbf{x}}=(x_{1},x_{2})$ on the grid identify the square defined by the four points $(x_{1},x_{2})$, $(x_{1}+1,x_{2})$, $(x_{1},x_{2}+1)$, and $(x_{1}+1,x_{2}+1)$. For example, when we say that the agent is at ${\mathbf{x}}=(x_{1},x_{2})$ we mean that the agent can be anywhere in the above square.

At each time instance, the agent can take $4$ actions: North, East, West, and South. Without loss of generality, we assume that the agent can only move to a neighboring grid point. The destination of the agent is based on its current position and the action. We assume that the agent movements are subject to uncertainty, which can cause the agent’s intended next position to shift to a point adjacent to that position. For example, the agent in state $(x_{1},x_{2})$ taking action North will enter state $(x_{1},x_{2}+1)$ with probability $0.8$ (the intended next position), but can enter states $(x_{1}-1,x_{2})$ or $(x_{1}+1,x_{2})$ with probability $0.1$, respectively. At the boundary points of the grid, the agent bounces against the “wall” in the opposite direction with its position unchanged.

The environment contains points with associated rewards. Specifically, as showed in Fig. 1, squares (waypoints) labeled $1$ and $36$ have associated reward equal to $1$ and squares (waypoints) labeled $6$ and $31$ have associated reward equal to $20$. The reward $r_{\mathbf{x}}$ of a waypoint ${\mathbf{x}}$ “spreads” on the grid according to a Gaussian function. Specifically, the immediate reward at point ${\mathbf{y}}$ due to the reward waypoint ${\mathbf{x}}$ is given by

[TABLE]

and is constant for all actions. Summing over all reward waypoints ${\mathbf{x}}$, the immediate reward at some point ${\mathbf{y}}$ is given by

[TABLE]

where we assume that a point ${\mathbf{x}}$ with no associated reward satisfies $r_{\mathbf{x}}=0$. The one step reward function induced by the four reward waypoints of Fig. 1 is shown in Fig. 2.

The agent (robot) entering some state (position) ${\mathbf{y}}$ collects reward equal to the immediate reward $f({\mathbf{y}})$. The objective of the agent is to navigate on the grid so as to maximize the long-term average reward collected.

The key step of efficient learning is to define appropriate features we will use to represent the agent’s policy. The way of selecting features in this paper is inspired by spline interpolation [23]. We define $36$ waypoints on the grid, shown in Fig. 1. For a state-action pair $({\mathbf{x}},a)$, we set ${\mathbf{y}}$ to be the intended next state and define the features

[TABLE]

where $f_{i}(\cdot)$ is as defined as

[TABLE]

and ${\mathbf{x}}_{i}$ is the location of the $i$th waypoint.

7.2 Simulation and Learning Performance

For the MDP we have introduced, we use value iteration [2] to find the best policy. We generate independent state-action samples according to this policy. Then, we estimate the policy using the above samples according to the logistic regression algorithm discussed in Section 3. We largely follow the style of [15] in presenting our numerical results. We compare average rewards from $3$ different policies with respect to the number of features and the number of samples as follows:

Target policy: We choose the policy obtained by the value iteration algorithm [2] as the target policy. It is used to generate samples for learning. 2. 2.

$\ell_{1}$-regularized policy: The RSP trained using the Algorithm in Section 3. 3. 3.

Unregularized policy: The RSP trained using logistic regression in Section 3, but without the $\ell_{1}$ constraint on the parameter vector $\boldsymbol{\theta}$ (cf. problem (3)). 4. 4.

Greedy policy: The agent takes the action with the largest expected next step reward, i.e., the local reward feature is the only consideration. This policy is used as a baseline.

We randomly sample the state-action pairs generated by the optimal policy and use these samples to form a training set to be used in learning an RSP (as in Section 3). This process is repeated $100$ times. The average rewards of all policies we considered as a function of the number of samples used for training are displayed in Fig. 3. As shown in Fig. 3, the $\ell_{1}$-regularized policy performs closely to the target policy, even with a very small number of training samples. Not surprisingly, both the $\ell_{1}$-regularized and the unregularized policy perform almost as well as the optimal policy when the number of training samples grows large. For most runs, the policies we learn (both regularized and unregularized) perform better than the greedy policy.

It is interesting that when the number of samples is small, the regularized policy performs better. This is because the unregularized policy tends to overfit to the training samples while the regularized policy seeks to learn fewer parameters and thus ends up learning them better, in a way that generalizes well out-of-sample. In sum, this simulation example supports the effectiveness of our algorithm.

8 Conclusion

We considered the problem of learning a policy in a Markov decision process using the state-action samples associated with the policy. We focused on a Boltzmann-type policy that is characterized by feature vectors associated with each state-action pair and a parameter that is sparse.

To learn the policy, we used $\ell_{1}$-regularized logistic regression and showed that a good generalization error bound also guarantees a good bound on the regret, defined as the difference between the average reward of the estimated policy and the target policy. Our results suggest that one can estimate an effective policy using a training set of size proportional to the logarithm of the number of features used to represent the policy.