A General Markov Decision Process Framework for Directly Learning Optimal Control Policies

TL;DR

This paper introduces a new reinforcement learning framework based on a generalized Markov decision process that directly learns optimal control policies, extending classical methods and demonstrating improved empirical performance.

Contribution

It proposes a novel MDP framework that generalizes policy definitions and supports direct learning of optimal control policies, including convergence proofs and empirical validation.

Findings

Demonstrates convergence of Q-learning within the new framework

Shows significant empirical benefits over traditional RL methods

Extends Bellman operator to support broader control paradigms

Abstract

We consider a new form of reinforcement learning (RL) that is based on opportunities to directly learn the optimal control policy and a general Markov decision process (MDP) framework devised to support these opportunities. Derivations of general classes of our control-based RL methods are presented, together with forms of exploration and exploitation in learning and applying the optimal control policy over time. Our general MDP framework extends the classical Bellman operator and optimality criteria by generalizing the definition and scope of a policy for any given state. We establish the convergence and optimality-both in general and within various control paradigms (e.g., piecewise linear control policies)-of our control-based methods through this general MDP framework, including convergence of $Q$-learning within the context of our MDP framework. Our empirical results demonstrate and quantify the significant benefits of our approach.

Tables

Table 1. Table 1: Average Lunar Lander Scores Over 1000 1000 1000 Episodes Across 50 50 50 Trials for Instances of Our Control-Based RL Approach and Classical Q 𝑄 Q -Learning with the Bellman Operator, including 95 95 95 % Confidence Intervals, During Testing Phase.

Control Model	Linear	PWL Sym ($M=2$)	PWL Sym ($M=4$)	Nonlinear	Q-learning
Score	216.05 $\pm$ 0.27%	$116.26$ $\pm 1.43\%$	$135.41$ $\pm 0.94\%$	$168.16$ $\pm 0.56\%$	$-203.21$ $\pm 0.51\%$

Table 2. Table 2: Average Mountain Car Scores Over 1000 1000 1000 Episodes Across 50 50 50 Trials for Instances of Our Control-Based RL Approach and Classical Q 𝑄 Q -Learning with the Bellman Operator, including 95 95 95 % Confidence Intervals, During Testing Phase.

Control Model	Linear	Nonlinear	Q-learning
Score	-105.72 $\pm$ 0.11%	-105.48 $\pm$ 0.11%	-129.68 $\pm$ 0.21%

Table 3. Table 3: Average Cart Pole Scores Over 1000 1000 1000 Episodes Across 50 50 50 Trials for Instances of Our Control-Based RL Approach and Classical Q 𝑄 Q -Learning with the Bellman Operator, including 95 95 95 % Confidence Intervals, During Testing Phase.

Controller/policy	Linear	Nonlinear	Q-learning
Reward	200.0 $\pm$ 0%	200.0 $\pm$ 0%	189.2 $\pm$ 0.24%

Equations

$$\underset{a_1,a_2,\dots }{\min }{\mathbb{E}}_{\mathbb{P}}\left[\sum _{t=0}^{\infty }{\gamma }^{t}c(x_t,a_t,x_{t+1})\right],\text{mbox}s.t.x_{t+1}=f(x_t,a_t),$$

$$\underset{a_1,a_2,\dots }{\max }{\mathbb{E}}_{\mathbb{P}}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t,a_t,x_{t+1})\right],\text{mbox}s.t.x_{t+1}=f(x_t,a_t),$$

$$a(y)=By+b\text{mbox}anda(y)=By+C{y}^{T}Dy+b,$$

$$\Vert {\mathbf{T}}_{\nu }{q}_1^{\mathcal{F}}-{\mathbf{T}}_{\nu }{q}_2^{\mathcal{F}}{\Vert }_{\infty }=$$

$$\le$$

$$q_{\ast }^{\mathcal{F}}(x,C)=\sum _{y\in \mathbb{X}}{P}_{C(x)}(x,y)[r(x,C(x),y)+\gamma \underset{D\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_{\ast }^{\mathcal{F}}(y,D)].$$

$$q_{t+1}^{\mathcal{F}}(x_t,C_t)=q_t^{\mathcal{F}}(x_t,C_t)+{\alpha }_t(x_t,C_t)\left[r_t+\gamma \underset{C\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_t^{\mathcal{F}}(x_{t+1},C)-q_t^{\mathcal{F}}(x_t,C_t)\right],$$

$$V^{\ast }(x)$$

$$V_i$$

$$\dot{\mathbf{x}}=f(\mathbf{x},t)$$

$${\mathbf{x}}_{n+1}=f({\mathbf{x}}_n,n)$$

$$\mathbf{u}=\mathbf{B}\mathbf{x}$$

$$Df({\mathbf{x}}_0,t)+\mathbf{B}$$

$$({\mathbf{T}}_{\nu }{q}^{\mathcal{F}})(x,C)=\sum _{y\in \mathbb{X}}{P}_{C(x)}(x,y)[r(x,C(x),y)+\gamma \underset{D\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}^{\mathcal{F}}(y,D)].$$

$$q_1^{\mathcal{F}}(x,C)\in \mathbb{Q}(\mathbb{X}\times \mathcal{F})\text{mbox}and{q}_2^{\mathcal{F}}(x,C)\in \mathbb{Q}(\mathbb{X}\times \mathcal{F}),$$

$$\Vert {\mathbf{T}}_{\nu }{q}_1^{\mathcal{F}}-{\mathbf{T}}_{\nu }{q}_2^{\mathcal{F}}{\Vert }_{\infty }=$$

$$\displaystyle\left.\qquad\qquad\qquad\qquad\qquad\quad-r(x,C(x),y)-\gamma\sup_{D_{2}\in{\mathcal{F}}}{{\mathbf{N}}}_{\nu}q^{{\mathcal{F}}}_{2}(y,D_{2})\Big{]}\right|$$

$$=$$

$$\le$$

$$=$$

$${\mathbf{N}}_{\nu }(q^{\mathcal{F}}(x,C))={\int }_{\nu (x)}{q}^{\mathcal{F}}(y,C)d{\mu }_x(y)$$

$$\Vert {\mathbf{N}}_{\nu }{q}_1^{\mathcal{F}}-{\mathbf{N}}_{\nu }{q}_2^{\mathcal{F}}{\Vert }_{\infty }\le \Vert \underset{y\in \nu (x)}{\sup }|q_1^{\mathcal{F}}(y,C)-q_2^{\mathcal{F}}(y,C)|{\Vert }_{\infty }\le \Vert q_1^{\mathcal{F}}-q_2^{\mathcal{F}}{\Vert }_{\infty }.$$

$$q_{\ast }^{\mathcal{F}}(x,C)=\sum _{y\in \mathbb{X}}{P}_{C(x)}(x,y)[r(x,C(x),y)+\gamma \underset{D\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_{\ast }^{\mathcal{F}}(y,D)].$$

$$q_{t+1}^{\mathcal{F}}(x_t,C_t)=q_t^{\mathcal{F}}(x_t,C_t)+{\alpha }_t(x_t,C_t)\left[r_t+\gamma \underset{C\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_t^{\mathcal{F}}(x_{t+1},C)-q_t^{\mathcal{F}}(x_t,C_t)\right],$$

$$q_t^{\mathcal{F}}(x_t,C_t)\text{mbox}and{r}_t+\gamma \underset{C\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_t^{\mathcal{F}}(x_{t+1},C),$$

$$q_{t+1}^{\mathcal{F}}(x_t,C_t)=\left(1-{\alpha }_t(x_t,C_t)\right)q_t^{\mathcal{F}}(x_t,C_t)+{\alpha }_t(x_t,C_t)\left[r_t+\gamma \underset{C\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_t^{\mathcal{F}}(x_{t+1},C)\right].$$

$${\Delta }_{t+1}(x_t,C_t)=\left(1-{\alpha }_t(x_t,C_t)\right){\Delta }_t(x_t,C_t)+{\alpha }_t(x_t,C_t)\left[r_t+\gamma \underset{C\in \mathcal{F}}{\sup }{\mathbf{N}}_{\nu }{q}_t^{\mathcal{F}}(x_{t+1},C)-q_{\ast }^{\mathcal{F}}\right].$$

$$F_t(x,C):=r(x,C(x),X(x,C))+\gamma \underset{D\in \mathcal{F}}{\max }{q}_t^{\mathcal{F}}(y,D)-q_{\ast }^{\mathcal{F}}(x,C),$$

$$|\mathbb{E}[F_t(x,C)|P_t]|$$

$$=|{\mathbf{T}}_{\nu }{q}_t^{\mathcal{F}}(x,C)-q_{\ast }^{\mathcal{F}}(x,C)|=|{\mathbf{T}}_{\nu }{q}_t^{\mathcal{F}}(x,C)-{\mathbf{T}}_{\mu }{q}_{\ast }^{\mathcal{F}}(x,C)|\le \gamma \Vert {\Delta }_t{\Vert }_{\infty }.$$

$$\mathbb{V}\text{mbox}ar[F_t(x,C)|P_t]$$

$$=\mathbb{V}\text{mbox}ar\left[r(x,C,X(x,C))+\gamma \underset{D\in \mathcal{F}}{\max }{q}_t^{\mathcal{F}}(y,D)|P_t\right]\le \kappa (1+\Vert {\Delta }_t{\Vert }_{\infty }^2),$$

Full text

1 Introduction

Over the past many years, reinforcement learning (RL) has proven to be very successful in solving a wide variety of learning and decision making problems. This includes problems related to game playing (e.g., [22, 23]), bicycle riding (e.g., [14]), and robotic control (e.g., [15]). Many different RL approaches, with varying levels of success, have been developed to address these problems [9, 21, 20].

Among these different approaches, model-free RL has been demonstrated to solve various problems without any prior knowledge (e.g., [14, 11]). Such model-free approaches, however, often suffer from high sample complexity that can require an inordinate amount of samples for some problems which can be prohibitive in practice, especially for problems limited by time or other constraints. Model-based RL has been demonstrated to significantly reduce sample complexity and shown to outperform model-free approaches for various problems (e.g., [5, 10]). Such model-based approaches, however, can often suffer from the difficulty of learning an appropriate model and from worse asymptotic performance than model-free approaches due to model bias from inherently assuming the learned system dynamics model accurately represents the true system environment (e.g., [2, 18, 17]).

In this paper we propose a novel form of RL that seeks to directly learn an optimal control model for a general underlying (unknown) dynamical system and to directly apply the corresponding optimal control from the model. This is in strong contrast to many traditional model-based RL methods that, after learning the system dynamics model which is often of high complexity and dimensionality, then use this system dynamics model to compute an optimal solution of a corresponding dynamic programming problem, often applying model predictive control (e.g., [13]). Our control-based RL approach instead learns the parameters of an optimal control model, often of lower complexity and dimensionality, from which the optimal solution is directly obtained. Furthermore, we establish that our control-based RL approach converges to an optimal solution analogous to model-free RL approaches while eliminating the problems of model bias in traditional model-based RL approaches.

We also introduce a general Markov decision process (MDP) framework that extends the classical Bellman operator and optimality criteria by generalizing the definition and scope of a policy for any given state and that provides theoretical support for our general control-based RL approach. Within this framework, we establish results on convergence with respect to (w.r.t.) both a contraction operator and a corresponding form of $Q$-learning, and then establish results on various aspects of optimality and optimal policies. To the best of our knowledge, this is the first proposal and analysis of a general control-based RL approach together with theoretical support from an extended MDP framework, both of which should be exploited to a much greater extent in the RL literature.

We first present our general control-based RL approach that directly learns the parameters of the optimal control models, including the corresponding convergence result. Our general MDP framework is then presented together with the corresponding theoretical results on convergence and optimality. Lastly, we present empirical results for a few classical problems in the OpenAI Gym framework [4] that demonstrate and quantify the significant benefits of our general approach over existing algorithms. We will release on GitHub the corresponding python code and make it publically available.

We refer to the Appendices in general for detailed proofs, additional technical details and results.

2 Control-Based Reinforcement Learning

Consider a discrete-state, discrete-time RL framework defined over a set of states ${{\mathbb{X}}}$, a set of actions ${{\mathbb{A}}}$, a transition probability kernel $\mathbb{P}$, a cost function ${c}$ or a reward function ${r}$ mapping state-action pairs to a bounded subset of ${\mathbb{R}}$, and a discount factor $\gamma\in[0,1)$. Let $\mathbb{E}_{\mathbb{P}}$ denote expectation w.r.t. the probability kernel $\mathbb{P}$. Then the discounted infinite-horizon stochastic dynamic programming (DP) formulation for the corresponding RL problems of interest can be expressed as

[TABLE]

where $x_{t}\in{{\mathbb{X}}}$ represents the state of the system, $a_{t}\in{{\mathbb{A}}}$ represents the control action decision variable, $f(\cdot,\cdot)$ represents the evolution function of the dynamical system characterizing the system state given the previous state and the taken action together with unknown uncertainty, and $c(\cdot,\cdot,\cdot)$ represents a cost-based objective function of both the system states and control action. Alternatively, the stochastic DP formulation associated with the RL problem of interest can be expressed as

[TABLE]

where $r(\cdot,\cdot,\cdot)$ represents a reward-based objective function of both the system states and control action, with all other variables and functions as described above.

We note that (1) and (2) can represent a wide variety of RL problems and corresponding stochastic DP problems based on the different forms taken by the evolution function $f(\cdot,\cdot)$, together with the transition probability kernel $\mathbb{P}$. For example, a linear system dynamics model results when $f(\cdot,\cdot)$ takes the form of linear transformations. The function $f(\cdot,\cdot)$ can also characterize the discretized evolutionary system dynamics governed by (stochastic) differential equations or (stochastic) partial differential equations. In addition, the cost function $c(\cdot,\cdot,\cdot)$ and reward function $r(\cdot,\cdot,\cdot)$ are also allowed to take on various general forms, and thus can represent any combination of cumulative and terminal costs or rewards, respectively. Both formulations (1) and (2) can also be analogously defined in continuous time. On the other hand, most classical RL formulations assume the dynamics evolve in discrete time. When the underlying system is based on a continuous-time model, then a discretization operator such as forward Euler discretization can be used to generate the discrete time samples.

Traditional model-based RL methods seek to first learn the system dynamics model of high complexity and dimensionality, and then incur the additional overhead of computing a solution to (1) or (2) to obtain an approximation of the optimal control policy w.r.t. the learned system dynamics model. In strong contrast, we take an optimal control-theoretic perspective and seek opportunities to directly learn an optimal control model of lower complexity and dimensionality for a general underlying dynamical system, and then directly apply the corresponding optimal control policy from the model.

Consider a control policy with a linear or simple nonlinear form, respectively given by

[TABLE]

where $y=K(x)$ is the output process representing a measurement of the state variable (which can be linear or nonlinear), $B,C,D,b$ are matrices or vectors with the proper dimensions, and $T$ denotes the transpose operator. We then need to learn $B,b$ in the case of the linear control model, and learn $B,C,D,b$ in the case of the simple nonlinear control model, based on the RL sample measurements satisfying the corresponding expressions above. Various forms of RL can be applied to determine the best parameters for $B,C,D,b$. In particular, we can exploit the low complexity and dimensionality of learning the parameters of the optimal control model, especially relative to the high complexity and dimensionality of learning the system dynamics model, to solve the corresponding optimization problem after a relatively small number of sample measurements. Hence, the learning problem is reduced to solving a relatively small stochastic optimization problem in which uncertainty has to be sampled. Many different algorithms can be deployed to solve these optimization problems and directly learn the desired control policy based on forms of exploration and exploitation.

More generally, our approach straightforwardly extends in an analogous manner to much more complicated nonlinear controllers, as well as piecewise linear and piecewise nonlinear controllers that can be considered as higher-order approximations of the optimal controller.

Our general control-based approach can be exploited together with many different forms of RL. We next summarize a generic simplified version of our approach.

Identify the reduced complexity and dimensionality control model based on the dimensions of the state and action vectors, possibly exploiting additional prior knowledge. 2. 2.

Initialize $B^{*}=B_{0},b^{*}=b_{0}$ (linear) or $B^{*}=B_{0},C^{*}=C_{0},D^{*}=D_{0},b^{*}=b_{0}$ (nonlinear). 3. 3.

Execute the system for the $e$-th episode of the RL task using the current control-model parameters $B_{e},b_{e}$ (linear) or $B_{e},C_{e},D_{e},b_{e}$ (nonlinear), thus generating a sequence of state, action, reward tuples. 4. 4.

When the episode yields is an improvement in the total reward, update the control-model parameters $B^{*}=B_{e},b^{*}=b_{e}$ (linear) or $B^{*}=B_{e},C^{*}=C_{e},D^{*}=D_{e},b^{*}=b_{e}$ (nonlinear). 5. 5.

Identify an alternative candidate for the control model $B_{e+1},b_{e+1}$ (linear) or $B_{e+1},C_{e+1},D_{e+1},b_{e+1}$ (nonlinear), based on one of numerous available options. 6. 6.

Increment $e$ and repeat from Step $3$ until $B^{*},b^{*}$ or $B^{*},C^{*},D^{*},b^{*}$ satisfy some criteria.

Prior knowledge (e.g., expert opinion, mathematical models), when available, can also be exploited in Step $1$ to bound the degrees of freedom of the problem; in such cases, we simply take advantage of the additional prior knowledge and boundary conditions available in many real-world problems. Initial conditions can be determined from prior solutions of similar problems, determined mathematically from a simple model, or chosen randomly. Step $3$ consists of evaluating the total reward of each episode $e$. An alternative candidate for the control model can be identified in numerous ways based on various forms of exploration and exploitation. While the above generic simplified version of our control-based RL approach can exploit many different algorithms of interest, we focus in this paper on a general class of optimal search algorithms to efficiently and effectively learn the control models, for which we establish below asymptotic convergence. We further note that, although our discussion thus far has focused on the control matrices $B,b$ (linear) or matrices $B,C,D,b$ (nonlinear), our general approach includes consideration of partitioning the state space into $M\geq 1$ distinct regions and similarly partitioning the control matrices into submatrices $\{B_{m},b_{m}\}$ (linear) or submatrices $\{B_{m},C_{m},D_{m},b_{m}\}$ (nonlinear) such that the submatrices indexed by $m$ are employed when the system is in states composing region $m=1,\ldots,M$. This will be further discussed in subsequent sections.

W.r.t our general control-based RL approach, we establish the following theoretical result on the guaranteed convergence to optimal stabilizing feedback control matrices for the system.

Theorem 2.1**.**

Under the assumption that optimal control matrices exist, the general control-based RL approach above will asymptotically converge to a set of optimal control matrices.

In the next section we will consider additional theoretical support for our general control-based RL approach through a general MDP framework, involving convergence and optimality both in general and within various control paradigms. Before doing so, however, we briefly discuss an illustrative example. As one simple generic example of our general control-based RL approach, consider a (unknown) dynamical system model $\dot{x}=f(x)$ and further consider the existence of a linear control model ($B$,$b$) such that $\dot{x}=f(x)+Bx+b$ will solve the problem at hand; analogously consider the existence of a simple nonlinear (quadratic) control model ($B$,$C$,$D$,$b$) such that $\dot{x}=f(x)+Bx+Cx^{T}Dx+b$ solves the problem. To illustrate how the system dynamics model tends to have higher complexity and dimensionality than that of the optimal control model within the context of this simple example, consider the Lunar Lander problem of Section 4 where the system is nonlinear and the state space is of dimension $6$, implying that each linearized vector field is of size $6\times 6$, has $36$ elements, and depends on the current state. This is in comparison with only $2$ control dimensions (left/right and vertical), and thus the linear control matrix $B$ and vector $b$ (which do not depend on the state) are of size $6\times 2$ and $1\times 2$, respectively, having a total of only $14$ elements. Analogously, the simple nonlinear (quadratic) control uses the scalar $x_{1}^{2}$ in addition to the $6$-dimensional state, which results in a $7\times 2$ matrix and a $1\times 2$ vector, totaling $16$ elements. Moreover, by exploiting additional knowledge, it can be possible to further restrict the degrees of freedom of the optimal control model.

3 Markov Decision Process Framework

We now turn to present a general MDP framework that provides additional theoretical support for our control-based RL approach and, in doing so, extends the classical Bellman operator and optimality criteria by generalizing the definition and scope of a policy for any state. This includes consideration of convergence and optimality, both in general and within various control paradigms such as linear and piecewise linear control policies, and convergence of $Q$-learning within the context of our framework.

Consider a discrete-space, discrete-time discounted MDP denoted by $({{\mathbb{X}}},{{\mathbb{A}}},\mathbb{P},{r},\gamma)$, where ${{\mathbb{X}}}$, ${{\mathbb{A}}}$, $\mathbb{P}$, ${r}$ and $\gamma\in[0,1)$ are as defined above. Let ${{\mathbb{Q}}}({{\mathbb{X}}}\times{{\mathbb{A}}})$ denote the space of bounded real-valued functions over ${{\mathbb{X}}}\times{{\mathbb{A}}}$ with supremum norm. Furthermore, for any $Q\in{{\mathbb{Q}}}$, define $V(x):=\max_{a}Q(x,a)$ with the same definition used for variants such as $\hat{Q}\in{{\mathbb{Q}}}$ and $\hat{V}(x)$. For the state $x_{t}$ at time $t\in{\mathbb{Z}}$ in which action $a$ is taken, i.e., $(x_{t},a)\in{{\mathbb{X}}}\times{{\mathbb{A}}}$, denote by $\mathbb{P}(\cdot|x_{t},a)$ the conditional transition probability for the next state $x_{t+1}$ and precisely define $\mathbb{E}_{\mathbb{P}}:=\mathbb{E}_{x_{t+1}\sim\mathbb{P}(\cdot|x_{t},a)}$ to be the expectation w.r.t. $\mathbb{P}(\cdot|x_{t},a)$.

We next introduce two key ideas to this standard MDP framework, one extending the policy $\pi:{{\mathbb{X}}}\rightarrow{{\mathbb{A}}}$ associated with the Bellman operator to a family of such policies and the other extending the domain of these policies from a single state $x$ to a neighborhood of states associated with $x$. More formally, define ${\mathcal{F}}$ to be a family of functions $f:{{\mathbb{X}}}\rightarrow{{\mathbb{A}}}$ between the set of states and the set of actions. The family ${\mathcal{F}}$ can be considered as the set of policies for the MDP, among which we will search to find the best policies. It is evident that, in the case ${\mathcal{F}}$ contains the functions $f(x)=\operatorname*{\arg\max}_{a\in{{\mathbb{A}}}}q(x,a)$ for all $q(x,a)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{{\mathbb{A}}})$, then our introduction of the family ${\mathcal{F}}$ continues to correspond to the standard MDP framework where the Bellman equation and operator are instrumental in calculating the optimal policies. Furthermore, for each $Q$-function $q(x,a)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{{\mathbb{A}}})$, we define the function $q^{\mathcal{F}}:{{\mathbb{X}}}\times{\mathcal{F}}\rightarrow{\mathbb{R}}$ as $q^{\mathcal{F}}(x,C)=q(x,C(x))$. It is readily apparent that $q^{\mathcal{F}}(x,C)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})$.

Turning to the second key aspect of our general MDP framework, let $\nu:{{\mathbb{X}}}\rightarrow 2^{{{\mathbb{X}}}}$ be a function mapping a state to a set of states with $x\in\nu(x)$ that defines a neighborhood of interest for each state $x\in{{\mathbb{X}}}$. Further define a generic operator ${{\mathbf{N}}}_{\nu}:{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})\rightarrow{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})$ that produces a new $q^{\mathcal{F}}$ function value at any point $(x,C)$ from the neighborhood set $\{q^{\mathcal{F}}(z,C),z\in\nu(x)\}$. When $\nu(x)=\{x\}$, we fix ${{\mathbf{N}}}_{\nu}$ such that ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=q^{\mathcal{F}}(x,C)$. For general functions $\nu$, examples based on ${{\mathbf{N}}}_{\nu}$ as the maximum, minimum and average operator respectively render ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\max_{y\in v(x)}q^{\mathcal{F}}(y,C)$, ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\min_{y\in v(x)}q^{\mathcal{F}}(y,C)$, and ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\int_{v(x)}q^{\mathcal{F}}(y,C)d\zeta(y)$ with probability measure $\zeta(y)$ defined on $\nu(x)$; for the special case of $\zeta(y)$ as the point measure at $x$, then ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=q^{\mathcal{F}}(x,C)$ which is equivalent to the case $\nu(x)=\{x\}$. By definition, these notions of the neighborhood function $\nu$ and the associated operator ${{\mathbf{N}}}_{\nu}$ satisfy the following claim.

Claim 3.1**.**

For any $x,y\in{{\mathbb{X}}}$, if $\nu(x)=\nu(y)$, then ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)={{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(y,C)$ for any $C\in{\mathcal{F}}$.

In the remainder of this section, we derive and establish theoretical results within the context of our general MDP framework to support our control-based approach. We first consider convergence w.r.t. both a contraction operator and a corresponding form of $Q$-learning, followed by various aspects of optimality, and then integration of our control-based approach and our MDP framework.

3.1 Convergence

We first seek to show that an analog of the Bellman operator within our general MDP framework is a contraction mapping in supremum norm. More specifically, the $Q$-function is a fixed point of a contraction operator ${{\mathbf{T}}}_{\nu}$ that is defined, for any function $q^{\mathcal{F}}(x,C)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})$, to be $({{\mathbf{T}}}_{\nu}q^{\mathcal{F}})(x,C)=\sum_{y\in{{\mathbb{X}}}}P_{C(x)}(x,y)[r(x,C(x),y)+\gamma\sup_{D\in{\mathcal{F}}}{{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(y,D)]$. Then, for any two functions $q^{{\mathcal{F}}}_{1}(x,C)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})$ and $q^{\mathcal{F}}_{2}(x,C)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})$, we have

[TABLE]

where the first equality is by definition and straightforward algebra, the first inequality is due to multiple applications of the triangle inequality, and the remaining directly follow for well-behaved operators ${{\mathbf{N}}}_{\nu}$. Under appropriate conditions on ${{\mathbf{T}}}_{\nu}$ and with $\gamma\in(0,1)$, this establishes that the operator is a contraction in the supremum norm, and therefore ${{\mathbf{T}}}^{t}_{\nu}(q^{\mathcal{F}})$ converges to a unique fixed point of the contraction operator in the limit as $t\rightarrow\infty$, for any function $q^{\mathcal{F}}:{{\mathbb{X}}}\times{\mathcal{F}}\rightarrow{\mathbb{R}}$.

The above derivation highlights the properties of the generic operator ${{\mathbf{N}}}_{\nu}$ that render the desired contraction mapping and convergence of ${{\mathbf{T}}}_{\nu}^{t}(q^{\mathcal{F}})$. Returning to the examples above, we observe that, for ${{\mathbf{N}}}_{\nu}(q^{{\mathcal{F}}}(x,C))=\int_{\nu(x)}q^{\mathcal{F}}(y,C)d\mu_{x}(y)$ with a sub-probability measure $\mu_{x}$ on $\nu(x)$, the following inequality holds: $\|{{\mathbf{N}}}_{\nu}q^{{\mathcal{F}}}_{1}-{{\mathbf{N}}}_{\nu}q^{{\mathcal{F}}}_{2}\|_{\infty}\leq\|\sup_{y\in\nu(x)}\,|q^{{\mathcal{F}}}_{1}(y,C)-q^{{\mathcal{F}}}_{2}(y,C)|\,\|_{\infty}\leq\|q^{{\mathcal{F}}}_{1}-q^{{\mathcal{F}}}_{2}\|_{\infty}$. For this general ${{\mathbf{N}}}_{\nu}$, which includes as special cases the maximum, minimum and average operators, we have with $\gamma\in(0,1)$ that ${{\mathbf{T}}}_{\nu_{t}}$ is a contraction and that ${{\mathbf{T}}}_{\nu_{t}}^{t}(q^{\mathcal{F}})$ converges to a unique fixed point as $t\rightarrow\infty$. Here $\nu_{t}$ is the neighborhood function for iteration $t$, where we allow the neighborhood to change over the iterative process. In all cases above, we observe that convergence of the contraction operator is to the unique fixed point $q_{*}^{\mathcal{F}}(x,C)$ which satisfies

[TABLE]

We next consider convergence of the $Q$-learning algorithm within the context of our general MDP framework. In particular, we focus on the following form of the classical Q-learning update rule [25]:

[TABLE]

for $0<\gamma<1$ and $0\leq\alpha_{t}(x_{t},C_{t})\leq 1$. Let $C_{t}$ be a sequence of controllers that covers all state-action pairs and $r_{t}$ the corresponding reward of applying $C_{t}$ to state $x_{t}$. We then have the following result.

Theorem 3.1**.**

Suppose ${{\mathbf{N}}}_{\nu}$ is defined as above such that ${{\mathbf{T}}}_{\nu}$ is a contraction. If $\sum_{t}\alpha_{t}=\infty$, $\sum_{t}\alpha_{t}^{2}<\infty$, and $r_{t}$ are bounded, then ${q}_{t}^{{\mathcal{F}}}$ converges to ${q}_{*}^{{\mathcal{F}}}$ as $t\rightarrow\infty$.

3.2 Optimality

We now consider aspects of optimality within our general MDP framework, which first enables us to derive an optimal policy from $q_{*}^{\mathcal{F}}(x,\cdot)$ as follows: $D_{{\mathcal{F}},{{\mathbf{N}}}_{\nu}}(x)=\operatorname*{\arg\max}_{C\in{\mathcal{F}}}{{\mathbf{N}}}_{\nu}q_{*}^{\mathcal{F}}(x,C)$. Define $V^{*}(x):=\sup_{C\in{\mathcal{F}}}{{\mathbf{N}}}_{\nu}q^{\mathcal{F}}_{*}(x,C)$. Then, from (4), we have

[TABLE]

Let us start by considering the optimal policy $D_{{\mathcal{F}},{{\mathbf{N}}}_{\nu}}(x)$ when $\nu(x)=\{x\}$, in which case (6) becomes $V^{*}(x)=\sup_{C\in{\mathcal{F}}}(\sum_{y\in{{\mathbb{X}}}}P_{C(x)}(x,y)[r(x,C(x),y)+\gamma V^{*}(y)]).$ This represents an optimality equation for the control policies in the family of functions ${\mathcal{F}}$. We therefore have the following.

Theorem 3.2**.**

Let $\nu(x)=\{x\},\forall x\in{{\mathbb{X}}}$. Then, $V^{*}(x)$ coincides with the value function of the MDP under the family ${\mathcal{F}}$ of control policies. Assuming ${\mathcal{F}}$ is sufficiently rich to include a function that assigns an optimal policy when in state $x,\forall x\in{{\mathbb{X}}}$, $V^{*}$ is the same as the optimal value function for the MDP. Furthermore, when ${\mathcal{F}}$ is the family of all (piecewise) linear control policies, $V^{*}$ represents the best objective function value that can be achieved by (piecewise) linear decision functions.

We next consider the case of general neighborhood functions $\nu$ and focus on the maximum operator ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\max_{y\in v(x)}q^{\mathcal{F}}(y,C)$. We observe that removal of ${{\mathbf{N}}}_{\nu}$ in (6) decreases the right hand side, and thus it is obviously true that $V^{*}(x)\geq\sup_{C\in{\mathcal{F}}}\sum_{y\in{{\mathbb{X}}}}P_{C(x)}(x,y)[r(x,C(x),y)+\gamma V^{*}(y)]$. This then leads to the following result.

Theorem 3.3**.**

Let the neighborhood function $\nu(x),\forall x\in{{\mathbb{X}}}$, be generally defined and let the neighborhood operator be the maximum operator, i.e., ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\max_{y\in v(x)}q^{\mathcal{F}}(y,C)$. Then $V^{*}(x)$ is an upper bound on the optimal value function of the MDP, for all $x\in{{\mathbb{X}}}$.

Let us now turn to consider optimality at a different level within our general MDP framework, where we suppose that the neighborhoods consist of a partitioning of the entire state space ${{\mathbb{X}}}$ into $M$ regions (consistent with Section 2), denoted by $\Omega_{1},\ldots,\Omega_{m}$, where $\nu(x)=\Omega_{m},\forall x\in\Omega_{m},m=1,2,\ldots,M$. From Claim 3.1, the right hand side of (6) has the same value for all states $x$ in the same neighborhood, and therefore we have $V^{*}(x)=V^{*}(y),\forall x,y\in\Omega_{m},m=1,2,\ldots,M$. We can write (6) as

[TABLE]

where $V_{i}$ represents the $V^{*}(x)$ for $x\in\Omega_{i}$. These equations can be further simplified depending on the different neighborhood operators, which leads to the following results for optimality and an optimal policy $D_{{\mathcal{F}},{{\mathbf{N}}}_{\nu}}(x)$ at the aggregate level in which the policies for state $x$ and state $y$ are the same whenever $x,y\in\Omega_{m},m=1,\ldots,M$ (also consistent with Section2).

Theorem 3.4**.**

Let the state space ${{\mathbb{X}}}$ be partitioned into disjoint sets $\Omega_{m}$ such that $\nu(x)=\Omega_{m},\forall x\in\Omega_{m}$, and let $p(x)$ be the function that maps a state to the region (partition) to which it belongs. For the average neighborhood operator ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\sum_{z\in\Omega_{p(x)}}\zeta(z)q^{\mathcal{F}}(z,C)$ with probability vector $\zeta(z)$ satisfying $\sum_{z\in\Omega_{m}}\zeta(z)=1$, $m=1,2,\ldots,M$, the corresponding version of (7) is the Bellman equation that produces the optimal value function for the MDP at the aggregate level. Furthermore, for the maximum neighborhood operator ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\max_{y\in v(x)}q^{\mathcal{F}}(y,C)$, the corresponding version of (7) is the Bellman equation that produces the optimal value function for the MDP at the aggregate level.

3.3 Control-Based Approach and MDP Framework

Lastly we consider combinations of our general control-based RL approach together with our general MDP framework. Theorem 2.1 guarantees that our control-based RL approach can asymptotically converge to a set of optimal control matrices, under the assumption of the existence of such optimal control matrices. This holds for the case of linear or piecewise linear control models $\{(B_{m},b_{m})\}$ and nonlinear or piecewise nonlinear (quadratic) control models $\{(B_{m},C_{m},D_{m},b_{m})\}$. When these control models contain an optimal policy, then Theorems 3.2 and 3.4 provide support for the optimality of the results and Theorem 2.1 ensures convergence to an optimal solution. On the other hand, if the class of control models of interest do not contain any optimal policies, then Theorem 3.3 establishes that an asymptotic analysis of these control models can lead to an upper bound on the optimal value function, thus providing theoretical support for such an approach as a good approximation.

Now consider an even tighter integration between our general control-based RL approach and our general MDP framework. Suppose that we start with a coarse partitioning of the state space, say $M=1$ and thus $\nu(x)=\nu(y),\forall x,y\in{{\mathbb{X}}}$. Let us focus on the case of linear control models, with the understanding that more general control models can be employed in an analogous manner. We iteratively follow this instance of our general control-based RL approach from Section 2 for some number of episodes, either until reaching convergence according to Theorem 2.1 with support from Theorem 3.4 or until achieving a desired level of performance relatively close to doing so. Then we increase $M$, partitioning the state space more finely, and once again iteratively follow this instance of our general control-based RL approach with the piecewise linear control models $\{(B_{m},b_{m})\}$ for some number of episodes along the very same lines. Once again, our results from Theorems 2.1 and 3.4 continue to hold and provide theoretical support. Further proceeding in this manner, with increasing $M$ and decreasing neighborhood sizes, we eventually reach the situation where $\nu(x)=\{x\},\forall x\in{{\mathbb{X}}}$. Then, by endowing ${\mathcal{F}}$ with a sufficiently rich set of functions, Theorem 3.2 together with Theorem 3.1 provide the necessary theoretical support that establish the resulting solution to be optimal. The empirical results in the next section provide further support for such an approach.

4 Experimental Results

We consider $3$ problems from OpenAI Gym [4]  –  Lunar Lander, Mountain Car and Cart Pole  –  and present empirical results that compare the performance of our control-based RL approach (CBRL) against that of the classical $Q$-learning approach with the Bellman operator (QLBO). The state space of each problem is continuous, which is directly employed in CBRL. However, for QLBO, the space is discretized to a finite set of states where each dimension is partitioned into equally spaced bins and the number of bins depends on both the problem to be solved and the reference codebase. Specifically, these experiments were executed using the existing code found at [24, 1], exactly as is with the default parameter settings. The exploration and exploitation aspects of CBRL include an objective that applies the methods over $K$ episodes with random initial conditions and averages the $K$ rewards; this is then solved using differential evolution [19] to find a controller that maximizes the objective function. Under the assumption that the system dynamics are unknown and that (piecewise) linear or nonlinear state feedback control is sufficient to solve the problem, the goal becomes learning the parameters of the corresponding optimal control model. We consider a (unknown) dynamical system model $\dot{x}=f(x,t)$ and a control action $a$ that is a function of $\nu(x)$ with several simple forms for $a$. This includes the control models:  (i) Linear: $a=Bx+b$;  (ii) Piecewise Linear: $a=B_{p(x)}x+b_{p(x)}$ where $p(x)=1,\ldots,M$ is the partition index for state $x$;  (iii) Nonlinear: $a=Bx+b+g(x)$. In summary, our CBRL control models provide significantly better performance in both the training and testing phases for each of the Lunar Lander, Mountain Car and Cart Pole problems considered.

Lunar Lander.

This problem is discussed in [4]. In each state, characterized by an $8$-dimensional vector, there are $4$ possible discrete actions (left, right, vertical, no thrusters). The goal is to maximize the cumulative reward (score) comprising positive points for successful degrees of landing and negative points for fuel usage and crashing. For the QLBO experiments, the $6$ continuous state variables are each discretized into $4$ bins. For the CBRL experiments, we consider $4$ different simple control models: Linear; Piecewise-Linear (PWL) Symmetric with $M=2$, $B_{1}=B_{2}$ except for $[B_{1}]_{*,1}=-[B_{2}]_{*,1}$, where $[B]_{*,1}$ denotes the first column of matrix $B$, and $b_{1}=b_{2}$; Piecewise-Linear (PWL) Symmetric with $M=4$, $B_{1}=B_{4}$ except for $[B_{1}]_{*,1}=-[B_{4}]_{*,1}$, $b_{1}=b_{4}$, $B_{2}=B_{3}$ except for $[B_{2}]_{*,1}=-[B_{3}]_{*,1}$, and $b_{2}=b_{3}$; Nonlinear with $g(x)=cx_{1}^{2}$ for scalar $x_{1}^{2}$.

Figure 1a plots the score results, averaged over $50$ experimental trials, as a function of the number of episodes for CBRL under the $4$ simple control models and for QLBO during the training phase. We observe that QLBO consistently exhibits significantly negative average scores with a trend over many episodes that remains flat. In strong contrast, our CBRL with each of the simple control models finds control policies that achieve average scores above $+200$ within $20,000$ episodes with trends continuing to slightly improve. We further observe that all $4$ CBRL result curves are quite similar.

Table 1 (Appendix D.1) presents the average score over $1000$ episodes across the $50$ trials for CBRL under the $4$ simple control models and for QLBO during the testing phase, together with the corresponding 95% confidence intervals. We again observe that the best average scores are by far obtained under the simple CBRL control models, with Linear considerably above $+200$ and QLBO below $-200$, and that the confidence intervals for all cases are quite small. We further observe the performance orderings among the simple CBRL control models with Linear providing the best results, followed by Nonlinear, PWL Symmetric ($M=4$), and PWL Symmetric ($M=2$). Note that the Linear and PWL Symmetric ($M=2$) controllers have the same number of parameters and they also provide the highest rewards during training; while the other CBRL controllers eventually approach these levels of reward performance, they do so more slowly due to the increased degrees of freedom (parameters). For the testing results, the CBRL controllers that provide better reward performance also have smaller confidence intervals, suggesting that they are closer to an optimal controller. Furthermore, the Linear controller provides the best performance, suggesting that the Nonlinear and PWL controllers require further searching of the decision space in the training phase to identify the corresponding optimal controller for the testing phase.

Mountain Car.

This problem is discussed in [12]. In each state, characterized by a $2$-dimensional vector, there are $3$ possible actions (forward, backward, neutral). The goal is to maximize the score representing the negation of the number of time steps needed to solve the problem (i.e., minimize number of steps to solve) over episodes of up to $200$ steps. For the QLBO experiments, the state space is discretized into a $40\times 40$ grid. For the CBRL experiments, we consider $2$ simple control models: Linear; Nonlinear with $g(x)$ as a $2$nd order multivariate polynomial on the state variables.

Figure 1b plots the score results, averaged over $50$ experimental trials, as a function of the number of episodes for CBRL under the $2$ simple control models and for QLBO during the training phase. We observe that the average scores for QLBO continually hover quite close to $-200$, as illustrated in the inset plot for the first $10,000$ episodes, implying that this algorithm is very rarely successful in solving the problem over a very large number of episodes across the $50$ trials. In strong contrast, the average scores for CBRL with the simple Linear and Nonlinear control models both follow a similar path towards $-110$, which is the point at which the problem is considered to be solved.

Table 2 (Appendix D.2) presents the average score over $1000$ episodes across the $50$ trials for CBRL under the $2$ simple control models and for QLBO during the testing phase, together with the corresponding 95% confidence intervals. We again observe that the best average scores are by far obtained under the simple CBRL control models, with Linear and Nonlinear both solving the problem and QLBO still far away, and that the confidence intervals for all cases are quite small. We further observe the performance orderings among the simple CBRL control models with Nonlinear slightly outperforming Linear, but the differences are very small. While the Nonlinear controller outperforms the Linear controller during both the training and testing phases, both simple CBRL control models perform extremely well, they completely solve the problem, and they significantly outperform QLBO.

Cart Pole.

This problem is discussed in [3]. In each state, characterized by a $4$-dimensional vector, there are $2$ possible discrete actions (push left, push right). The goal is to maximize the score representing the number of steps that the cart pole stays upright before either falling over or going out of bounds. For the QLBO experiments, the position and velocity are discretized into $8$ bins whereas the angle and angular velocity are discretized into $10$ bins. For the CBRL experiments, we consider $2$ different simple control models: Linear; Nonlinear with $g(x)$ as a $2$nd order multivariate polynomial on the state variables. With a score of $200$, the problem is considered solved and the simulation ends.

Figure 1c plots the score results, averaged over $50$ experimental trials, as a function of the number of episodes for CBRL under the $2$ simple control models and for QLBO during the training phase. We observe that CBRL, with both the simple Linear and Nonlinear control models, quickly finds an optimal control policy that solves the problem within a few hundred episodes, whereas QLBO continues to oscillate well below the maximal score of $200$ even for this relatively simple problem.

Table 3 (Appendix D.3) presents the average score over $1000$ episodes across the $50$ trials for CBRL under the $2$ simple control models and for QLBO during the testing phase, together with the corresponding 95% confidence intervals. We again observe that the best average scores are by far obtained under the simple CBRL control models, with Linear and Nonlinear both solving the problem while having confidence intervals of essentially zero and with QLBO providing poorer performance and still a considerable distance away from optimal while having a very small confidence interval. In addition, the Linear controller with less degrees of freedom reaches higher scores more quickly than the Nonlinear controller, but both controllers reach a score of $200$ after a few hundred episodes.

Broader Impact

Our contributions are in the areas of reinforcement learning (RL) and Markov decision processes (MDP), and as such have the potential of impacting a very broad spectrum of applications in business, engineering and science. The great success of RL to solve a wide variety of learning and decision problems can be further enhanced through the approach and results of the paper. Since our approach and results improve the sample complexity problem of RL, we believe there is the potential of enabling more applications to learn faster and with less data. In addition, we plan to include the main approaches and results of this paper as part of the curriculum in graduate courses.

Appendix A Generic Algorithm for General Control-Based Reinforcement Learning

Our general control-based RL approach can be summarized by the following generic simplified algorithm, using the notation described in Sections 2 and 3.

Appendix B Control-Based Reinforcement Learning

In this section we present additional technical details and results related to Section 2.

B.1 Proof of Theorem 2.1

First, consider the dynamics of a system given by

[TABLE]

for the continuous-time case or by

[TABLE]

for the discrete-time case. Suppose $f$ is Lipschitz continuous with Lipschitz constant $L$ and the goal of the task at hand is to ensure that the trajectory of the system converges toward an equilibrium point of the system ${\mathbf{x}}_{0}$. Lyapunov stability analysis and known results from time-varying linear systems [7] show that

[TABLE]

is a linear stabilizing feedback provided that

[TABLE]

has eigenvalues with negative real parts bounded away from the origin and slowly varying, where $Df(\cdot,\cdot)$ is the Jacobian matrix of $f(\cdot,\cdot)$. In particular, $-\alpha{\mathbf{I}}$ is a stabilizing feedback matrix where $\alpha>L$.

This then implies that an appropriate search on $\alpha$ will result in a stabilizing feedback. Such an optimal search can be performed to find feedback matrices that satisfy bounds, such as spectral bounds, using both gradient-based and gradient-free methods, such as local search, differential evolution, Bayesian optimization and genetic algorithms. The eigenvalue conditions show that a feedback matrix within a small ball around $B$ will still stabilize the system, and therefore such instances of our general control-based RL approach will guarantee to find a solution asymptotically given sufficient time to search the space, analogous to the guaranteed asymptotic convergence under the Bellman equation.

Appendix C Markov Decision Process Framework

In this section we present additional technical details and results related to Section 3.

C.1 Convergence

We first seek to show that an analog of the Bellman operator within our general MDP framework is a contraction mapping in supremum norm. More specifically, the $Q$-function is the fixed point of a contraction operator ${{\mathbf{T}}}_{\nu}$ that is defined, for any function $q^{\mathcal{F}}(x,C)\in{{\mathbb{Q}}}({{\mathbb{X}}}\times{\mathcal{F}})$, to be

[TABLE]

Then, for any two functions

[TABLE]

we have

[TABLE]

where the first equality is by definition, the second equality follows from straightforward algebra, the first and second inequalities are due to the triangle inequality, and the remaining directly follow for well-behaved operators ${{\mathbf{N}}}_{\nu}$. Under appropriate conditions on ${{\mathbf{T}}}_{\nu}$ and with $\gamma\in(0,1)$, this establishes that the operator is a contraction in the supremum norm, and therefore ${{\mathbf{T}}}^{t}_{\nu}(q^{\mathcal{F}})$ converges to a unique fixed point of the contraction operator in the limit as $t\rightarrow\infty$, for any function $q^{\mathcal{F}}:{{\mathbb{X}}}\times{\mathcal{F}}\rightarrow{\mathbb{R}}$.

The above derivation highlights the properties of the generic operator ${{\mathbf{N}}}_{\nu}$ that render the desired contraction mapping and convergence of ${{\mathbf{T}}}_{\nu}^{t}(q^{\mathcal{F}})$. Returning to the examples above, we observe that, for

[TABLE]

with a sub-probability measure $\mu_{x}$ on $\nu(x)$, the following inequality holds:

[TABLE]

For this general ${{\mathbf{N}}}_{\nu}$, which includes as special cases the maximum, minimum and average operators, we have with $\gamma\in(0,1)$ that ${{\mathbf{T}}}_{\nu_{t}}$ is a contraction and that ${{\mathbf{T}}}_{\nu_{t}}^{t}(q^{\mathcal{F}})$ converges to a unique fixed point in the limit as $t\rightarrow\infty$. Here $\nu_{t}$ is the neighborhood function for iteration $t$, where we allow the neighborhood to change over the iterative process. In all cases above, we observe that the fixed point $q_{*}^{\mathcal{F}}(x,C)$ of the contraction operator satisfies

[TABLE]

We next consider convergence of a form of the $Q$-learning algorithm within the context of our general MDP framework. In particular, we focus on the following form of the classical Q-learning update rule [25]:

[TABLE]

for $0<\gamma<1$ and $0\leq\alpha_{t}(x_{t},C_{t})\leq 1$. Let $C_{t}$ be a sequence of controllers that covers all state-action pairs and $r_{t}$ the corresponding reward of applying $C_{t}$ to state $x_{t}$. We then have the result in Theorem 3.1 whose proof is given in the next section.

C.2 Proof of Theorem 3.1

The proof is somewhat similar to the well-known proof of the convergence of the classical $Q$-learning update (see, e.g,, [8, 6]) within the context of our general MDP framework. We first rewrite (9) as a convex combination of

[TABLE]

where

[TABLE]

Define the difference between $q_{t}^{{\mathcal{F}}}$ and $q_{*}^{{\mathcal{F}}}$ as $\Delta_{t}:=q_{t}^{{\mathcal{F}}}-q_{*}^{{\mathcal{F}}}$, which satisfies

[TABLE]

Further define

[TABLE]

where $X(x,C)$ is a randomly sampled state from the MDP under an initial state $x$ and policy $C$, from which we obtain

[TABLE]

where $P_{t}=\{\Delta_{t},\Delta_{t-1},\cdots,F_{t-1},\cdots,\alpha_{t-1},\cdots\}$ stands for the past state and parameters [8]. Similarly, we have

[TABLE]

for some $\kappa>0$. The desired result then follows from Theorem 1 in [8].

C.3 Optimality

We now consider aspects of optimality within our general MDP framework, which first enables us to derive an optimal policy from $q_{*}^{\mathcal{F}}(x,\cdot)$ as follows:

[TABLE]

Define $V^{*}(x):=\sup_{C\in{\mathcal{F}}}{{\mathbf{N}}}_{\nu}q^{\mathcal{F}}_{*}(x,C)$. Then, from (8), we have

[TABLE]

Let us start by considering the optimal policy $D_{{\mathcal{F}},{{\mathbf{N}}}_{\nu}}(x)$ in (10) when $\nu(x)=\{x\}$, in which case (11) becomes

[TABLE]

This represents an optimality equation for the control policies in the family of functions ${\mathcal{F}}$. We therefore have the result in Theorem 3.2 whose proof is given in Section C.4.

We next consider the case of general neighborhood functions $\nu$ and focus on the maximum operator ${{\mathbf{N}}}_{\nu}q^{\mathcal{F}}(x,C)=\max_{y\in v(x)}q^{\mathcal{F}}(y,C)$. In this case, we observe that removal of ${{\mathbf{N}}}_{\nu}$ in (11) decreases the right hand side, and thus it is obviously true that

[TABLE]

This then leads to the result in Theorem 3.3 whose proof is given in Section C.5.

Let us now turn to consider optimality at a different level within our general MDP framework, where we suppose that the neighborhoods consist of a partitioning of the entire state space ${{\mathbb{X}}}$ into $M$ regions (consistent with Section 2), denoted by $\Omega_{1},\ldots,\Omega_{m}$, with $\nu(x)=\Omega_{m},\forall x\in\Omega_{m},m=1,2,\ldots,M$. From Claim 3.1, the right hand side of (11) has the same value for all states $x$ in the same neighborhood, and therefore we have $V^{*}(x)=V^{*}(y),\forall x,y\in\Omega_{m},m=1,2,\ldots,M$. We can express (11) as

[TABLE]

where $V_{i}$ represents the $V^{*}(x)$ for $x\in\Omega_{i}$ and $p(y)$ is the function that maps a state to the region (partition) to which it belongs. This can be equivalently written as

[TABLE]

These equations can be further simplified depending on the different neighborhood operators, which leads to the results in Theorem 3.4 for optimality and an optimal policy $D_{{\mathcal{F}},{{\mathbf{N}}}_{\nu}}(x)$ at the aggregate level where the policies for state $x$ and state $y$ are the same whenever $x,y\in\Omega_{m},m=1,\ldots,M$ (also consistent with Section2). The proof is provided in Section C.6.

C.4 Proof of Theorem 3.2

With $\nu(x)=\{x\},\forall x\in{{\mathbb{X}}}$, we know from (10) and (11) that the value function $V^{*}(x)$ satisfies (12), which represents an optimality equation for the control policies in the family of functions ${\mathcal{F}}$. Hence, $V^{*}(x)$ coincides with the value function of the MDP under the family ${\mathcal{F}}$ of control policies. Under the assumption that ${\mathcal{F}}$ is sufficiently rich to include a function that assigns optimal policy when in state $x,\forall x\in{{\mathbb{X}}}$ (such as the one determined by the Bellman operator but not restricted to this specific optimal policy), then this value function $V^{*}$ is the same as the optimal value function for the MDP. On the other hand, under the assumption that ${\mathcal{F}}$ is the family of all (piecewise) linear control policies, then this value function $V^{*}$ represents the best objective function value that can be achieved by (piecewise) linear decision functions.

C.5 Proof of Theorem 3.3

For an MDP within the context of our framework, the optimal value function that satisfies

[TABLE]

is well known (see, e.g., [16, Proposition 5.1, Chapter II]) to be the minimum value of all the functions $v^{\prime}(x)$ that satisfy

[TABLE]

Hence, (13) ensures that $V^{*}(x)$ is an upper bound on the optimal value function.

C.6 Proof of Theorem 3.4

With the state space ${{\mathbb{X}}}$ partitioned into disjoint sets $\Omega_{m}$ such that $\nu(x)=\Omega_{m},\forall x\in\Omega_{m}$, $m=1,\ldots,M$, let us first consider the average neighborhood operator

[TABLE]

where the probability vector $\zeta(z)$ satisfies $\sum_{z\in\Omega_{m}}\zeta(z)=1$, for all $m=1,2,\ldots,M$. We then have (14) taking the form

[TABLE]

and therefore

[TABLE]

Now consider a new MDP with states $1,2,\ldots,M$, action space ${\mathcal{F}}$, transition probabilities

[TABLE]

and rewards

[TABLE]

Hence (15) is the Bellman equation that produces the optimal value function for this new MDP, which corresponds to the original MDP at the aggregate level.

Under the same state space partitioning, let us next consider the maximum neighborhood operator

[TABLE]

We then have (14) taking the form

[TABLE]

and therefore

[TABLE]

Once again, consider a new MDP with states $1,2,\ldots,M$, action sets $\{\Omega_{i},{\mathcal{F}}\}_{i}$, transition probabilities

[TABLE]

and rewards

[TABLE]

where $z\in\Omega_{i}$ in both the transition probability and reward definitions. Hence (16) is the Bellman equation that produces the optimal value function for this new MDP, which corresponds to the original MDP at the aggregate level.

Appendix D Experimental Results

In this section we present additional material and results that complement those in the main paper.

We consider $3$ problems from OpenAI Gym [4]  –  namely Lunar Lander, Mountain Car and Cart Pole  –  and we present empirical results for each problem that compare the performance of our control-based RL approach against that of the classical $Q$-learning approach with the Bellman operator. The state space of each problem is continuous, which is directly employed in our control-based RL approach. However, for classical $Q$-learning with the Bellman operator, the space is discretized to a finite set of states where each dimension is partitioned into equally spaced bins and the number of bins depends on both the problem to be solved and the reference codebase. The exploration and exploitation aspects of our control-based RL approach include an objective that applies the methods over $K$ episodes with random initial conditions and averages the $K$ rewards; this is then solved using differential evolution [19] to find a controller that maximizes the objective function. Under the assumption that the system dynamics are unknown and that (piecewise) linear or nonlinear state feedback control is sufficient to solve the problem at hand, the goal then becomes learning the parameters of the corresponding optimal control model. We consider a (unknown) dynamical system model $\dot{x}=f(x,t)$ and a control action $a$ that is a function of the neighborhood $\nu(x)$ of the state $x$ with several simple forms for $a$. This includes (variants of) the control models:

Linear: $a=Bx+b$; 2. 2.

Piecewise Linear: $a=B_{p(x)}x+b_{p(x)}$ where $p(x)=1,\ldots,M$ is the partition index for state $x$; 3. 3.

Nonlinear: $a=Bx+b+g(x)$ where $g(x)$ is a general function of $x$.

Multiple experimental trials are run for each problem from the OpenAI Gym. For each problem, our experimental results under the classical $Q$-learning approach with the Bellman operator were obtained using the existing code found at [24, 1] exactly as is with the default parameter settings. This includes use of a standard, yet generic, form for the $Q$-learning algorithms. Specifically, for all $Q_{0}\in{{\mathbb{Q}}}$, $x\in{{\mathbb{X}}}$, $a\in{{\mathbb{A}}}$ and the Bellman operator ${{\mathbf{B}}}$, the sequence of action-value $Q$-functions is updated based on the following generic rule:

[TABLE]

where $\alpha_{k}$ is the learning rate for iteration $k$. At each timestep, (17) is iteratively applied to the $Q$-function at the current state and action.

We note that each of the algorithms from the OpenAI Gym implements a form of the $\epsilon$-greedy method (e.g., occasionally picking a random action or using a randomly perturbed $Q$-function for determining the action) to enable some form of exploration in addition to the exploitation-based search of the optimal policy using the $Q$-function. Our experiments were therefore repeated over a wide range of values for $\epsilon$, where we found that the relative performance trends of the different approaches did not depend significantly on the amount of exploration under the $\epsilon$-greedy algorithm. In particular, the same performance trends were observed over a wide range of $\epsilon$ values and hence we present results based on the default value of $\epsilon$ used in the reference codebase.

In short, as an overall general summary of the experimental results for each of the Lunar Lander, Mountain Car and Cart Pole problems considered, our control-based RL approach (CBRL) provides significantly better performance over that exhibited under the classical $Q$-learning approach with the Bellman operator (QLBO) in both training and testing phases.

D.1 Lunar Lander

This problem is discussed in Brockman et al. [4]. In each state, characterized by an $8$-dimensional state vector, there are $4$ possible discrete actions (left, right, vertical, no thrusters). The goal is to maximize the cumulative reward (score) comprising positive points for successful degrees of landing and negative points for fuel usage and crashing. For the QLBO experiments, the $6$ continuous state variables are each discretized into $4$ bins. For the CBRL experiments, we consider $4$ different simple control models:

Linear; 2. 2.

Piecewise-Linear (PWL) Symmetric with $M=2$, $B_{1}=B_{2}$ except for $[B_{1}]_{*,1}=-[B_{2}]_{*,1}$, where $[B]_{*,1}$ denotes the first column of matrix $B$, and $b_{1}=b_{2}$; 3. 3.

Piecewise-Linear (PWL) Symmetric with $M=4$, $B_{1}=B_{4}$ except for $[B_{1}]_{*,1}=-[B_{4}]_{*,1}$, $b_{1}=b_{4}$, $B_{2}=B_{3}$ except for $[B_{2}]_{*,1}=-[B_{3}]_{*,1}$, and $b_{2}=b_{3}$; 4. 4.

Nonlinear with $g(x)=cx_{1}^{2}$ for scalar $x_{1}^{2}$.

Figure 2 plots the score results, averaged over $50$ experimental trials, as a function of the number of episodes for CBRL under the $4$ simple control models and for QLBO during the training phase. We observe that QLBO consistently exhibits significantly negative average scores with a trend over many episodes that remains flat. In strong contrast, our CBRL with each of the simple control models finds control policies that achieve average scores above $+200$ within $20,000$ episodes with trends continuing to slightly improve. We further observe that all $4$ CBRL result curves are quite similar.

Table 1 presents the average score over $1000$ episodes across the $50$ trials for CBRL under the $4$ simple control models and for QLBO during the testing phase, together with the corresponding 95% confidence intervals. We again observe that the best average scores are by far obtained under the simple CBRL control models, with Linear considerably above $+200$ and QLBO below $-200$, and that the confidence intervals for all cases are quite small. We further observe the performance orderings among the simple CBRL control models with Linear providing the best results, followed by Nonlinear, PWL Symmetric ($M=4$), and PWL Symmetric ($M=2$). Note that the Linear and PWL Symmetric ($M=2$) controllers have the same number of parameters and they also provide the highest rewards during training; while the other CBRL controllers eventually approach these levels of reward performance during training, they do so more slowly due to the increased degrees of freedom (parameters). For the testing results, the CBRL controllers that provide better reward performance also have smaller confidence intervals, suggesting that they are closer to an optimal controller. Furthermore, the Linear controller provides the best performance, suggesting that the Nonlinear and PWL controllers require further searching of the decision space in the training phase to identify the corresponding optimal controller for the testing phase. In particular, one can imagine that PWL controllers (and piecewise Nonlinear controllers) under appropriate partitionings should provide superior performance over that of the Linear controller for problems such as Lunar Lander, but this requires better searching of the decision space in the training phase to identify the corresponding optimal PWL controller for the testing phase, possibly along the lines of Section 3.3.

D.2 Mountain Car

This problem is first discussed in [12]. In each state, characterized by a $2$-dimensional state vector, there are $3$ possible actions (forward, backward, neutral). The goal is to maximize the score representing the negation of the number of time steps needed to solve the problem (i.e., minimize number of steps to solve) over episodes of up to $200$ steps. For the QLBO experiments, the state space is discretized into a $40\times 40$ grid. For the CBRL experiments, we consider $2$ simple control models:

Linear; 2. 2.

Nonlinear with $g(x)$ as a second-order multivariate polynomial on the state variables.

Figure 3 plots the score results, averaged over $50$ experimental trials, as a function of the number of episodes for CBRL under the $2$ simple control models and for QLBO during the training phase. We observe that the average scores for QLBO continually hover within the range of $[-160,-200]$, implying that this algorithm is rarely successful in solving the problem over a very large number of episodes across the $50$ trials. In strong contrast, the average scores for CBRL with the simple Linear and Nonlinear control models both follow a similar path towards $-110$, which is the point at which the problem is considered to be solved. We further observe the noticeable performance improvements under the Nonlinear controller over the Linear controller during the training phase.

Table 2 presents the average score over $1000$ episodes across the $50$ trials for CBRL under the $2$ simple control models and for QLBO during the testing phase, together with the corresponding 95% confidence intervals. We again observe that the best average scores are by far obtained under the simple CBRL control models, with Linear and Nonlinear both solving the problem and QLBO still far away, and that the confidence intervals for all cases are quite small. We further observe the performance orderings among the simple CBRL control models with Nonlinear slightly outperforming Linear, but the differences are very small. While the Nonlinear controller outperforms the Linear controller during both the training and testing phases, both simple CBRL control models perform extremely well, they completely solve the problem, and they significantly outperform QLBO.

D.3 Cart Pole

This problem is discussed in Barto et al. [3]. In each state, characterized by a $4$-dimensional state vector, there are $2$ possible discrete actions (push left, push right). The goal is to maximize the score representing the number of steps that the cart pole stays upright before either falling over or going out of bounds. For the QLBO experiments, the position and velocity are discretized into $8$ bins whereas the angle and angular velocity are discretized into $10$ bins. For the CBRL experiments, we consider $2$ different simple control models:

Linear; 2. 2.

Nonlinear with $g(x)$ as a second-order multivariate polynomial on the state variables.

With a score of $200$, the problem is considered solved and the simulation ends.

Figure 4 plots the score results, averaged over $50$ experimental trials, as a function of the number of episodes for CBRL under the $2$ simple control models and for QLBO during the training phase. We observe that CBRL, with both the simple Linear and Nonlinear control models, quickly finds an optimal control policy that solves the problem within a few hundred episodes, whereas QLBO continues to oscillate well below the maximal score of $200$ even for this relatively simple problem.

Table 3 presents the average score over $1000$ episodes across the $50$ trials for CBRL under the $2$ simple control models and for QLBO during the testing phase, together with the corresponding 95% confidence intervals. We again observe that the best average scores are by far obtained under the simple CBRL control models, with Linear and Nonlinear both solving the problem while having confidence intervals of essentially zero and with QLBO providing poorer performance and still a considerable distance away from optimal while having a very small confidence interval. In addition, the Linear controller with less degrees of freedom reaches higher scores more quickly than the Nonlinear controller during the training phase, but both controllers reach a score of $200$ after a few hundred episodes.