Deep Reinforcement-Learning-based Driving Policy for Autonomous Road Vehicles

TL;DR

This paper introduces a reinforcement learning-based driving policy for autonomous vehicles on freeways, capable of operating in mixed traffic environments without relying on detailed environmental models.

Contribution

It presents one of the first reinforcement learning policies for mixed autonomous and manual traffic, avoiding assumptions about environment dynamics.

Findings

RL policy performs comparably to optimal control methods.

The policy is effective in realistic traffic scenarios.

Initial results show autonomous vehicle behavior impacts traffic flow.

Abstract

In this work the problem of path planning for an autonomous vehicle that moves on a freeway is considered. The most common approaches that are used to address this problem are based on optimal control methods, which make assumptions about the model of the environment and the system dynamics. On the contrary, this work proposes the development of a driving policy based on reinforcement learning. In this way, the proposed driving policy makes minimal or no assumptions about the environment, since a priori knowledge about the system dynamics is not required. Driving scenarios where the road is occupied both by autonomous and manual driving vehicles are considered. To the best of our knowledge, this is one of the first approaches that propose a reinforcement learning driving policy for mixed driving environments. The derived reinforcement learning policy, firstly, is compared against an optimal policy derived via dynamic programming, and, secondly, its efficiency is evaluated under realistic scenarios generated by the established SUMO microscopic traffic flow simulator. Finally, some initial results regarding the effect of autonomous vehicles' behavior on the overall traffic flow are presented.

Tables

Table 1. Table 1: The available actions of the autonomous vehicle.

Action #1:	Change lane to the left
Action #2:	Change lane to the right
Action #3:	Constant acceleration of $1m/s^2$
Action #4:	Constant acceleration of $2m/s^2$
Action #5:	Constant deceleration of $1m/s^2$
Action #6:	Constant deceleration of $2m/s^2$
Action #7:	Move on current lane with current speed

Table 2. Table 2: Driving behavior evaluation of the RL and DP driving policies, in terms of total number of collision and lane changes for 100 scenarios and percentage of time that the vehicle moves with its desired speed.

1 veh./8 sec.	Collisions	Lane changes	Desired speed (%)
DP policy	0	84	85
RL policy	0	81	73
1 veh./4 sec.
DP policy	0	127	83
RL policy	0	115	64
1 veh./2 sec.
DP policy	0	120	87
RL policy	0	108	62
1 veh./1 sec.
DP policy	0	70	72
RL policy	2	62	56

Table 3. Table 3: Driving behavior evaluation. SUMO default corresponds to the default SUMO configuration, while SUMO manual to the case where the behavior of the autonomous vehicle is the same as the manual driving vehicles.

Desired speed for slow vehicles 18m/s
	Collisions	Avg speed
RL policy with rules ($\sigma =0.0$)	0%	20.62
RL policy w/o rules ($\sigma =0.0$)	2%	20.71
SUMO default ($\sigma =0.0$)	0%	20.22
SUMO manual ($\sigma =0.0$)	0%	19.48
RL policy with rules ($\sigma =0.5$)	0%	20.08
RL policy w/o rules ($\sigma =0.5$)	3%	20.09
SUMO default ($\sigma =0.5$)	0%	19.57
SUMO manual ($\sigma =0.5$)	0%	19.05
Desired speed for slow vehicles 16m/s
	Collisions	Avg speed
RL policy with rules ($\sigma =0.0$)	0%	19.87
RL policy w/o rules ($\sigma =0.0$)	2%	20.04
SUMO default ($\sigma =0.0$)	0%	18.41
SUMO manual ($\sigma =0.0$)	0%	17.47
RL policy with rules ($\sigma =0.5$)	0%	19.81
RL policy w/o rules ($\sigma =0.5$)	4%	19.87
SUMO default ($\sigma =0.5$)	0%	17.67
SUMO manual ($\sigma =0.5$)	0%	17.26

Table 4. Table 4: Driving behavior evaluation with ad-hoc safety rules when different magnitudes of measurements errors are introduced.

5% Noise
	Collisions	Avg speed
RL policy with rules ($\sigma =0.0$)	0%	19.88
RL policy with rules ($\sigma =0.5$)	0%	19.84
10% Noise
	Collisions	Avg speed
RL policy with rules ($\sigma =0.0$)	0%	19.65
RL policy with rules ($\sigma =0.5$)	0%	19.59

Table 5. Table 5: Vehicle types present on the road.

	Percentage	Maximun Speed
Slow passenger vehicles	40%	16 m/s
Fast passenger vehicles	40%	25 m/s
Trucks	5%	14 m/s
Buses	5%	16 m/s
Motorcycles	10%	21 m/s

Table 6. Table 6: Driving behavior evaluation with ad-hoc safety rules when different types of vehicles occupy the road.

	Collisions	Avg speed
RL policy with rules ($\sigma =0.0$)	0%	19.74
RL policy with rules ($\sigma =0.5$)	0%	19.67

Table 7. Table 7: Driving behavior evaluation with ad-hoc safety rules rainy weather conditions.

	Collisions	Avg speed
RL policy with rules ($\sigma =0.5$)	0%	19.48

Table 8. Table 8: Effect of autonomous vehicles on the overall traffic flow.

	Avg speed	Improvement over 0%
Autonomous vehicles 0 %	15.32 m/s	0.0%
Autonomous vehicles 5 %	15.41 m/s	0.6%
Autonomous vehicles 10%	16.11 m/s	5.1%
Autonomous vehicles 20%	15.91 m/s	1.3%

Equations

$$R_t=\sum _{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }},$$

$$Q^{\ast }(s,a)=\underset{\pi }{\max }\mathbb{E}[R_t|s_t=s,a_t=a,\pi ].$$

$$Q^{\ast }(s,a)={\mathbb{E}}_{s_{t+1}}[r_t+\gamma \underset{a_{t+1}}{\max }{Q}^{\ast }(s_{t+1},a_{t+1})|s_t=s,a_t=a]$$

$$L_i({\theta }_i)={\mathbb{E}}_{s,a}[(\tilde{Q}(s,a;{\theta }_{i-1})-\tilde{Q}(s,a;{\theta }_i){)}^2],$$

$$L_i({\theta }_i)={\mathbb{E}}_{s,a}[(\tilde{Q}(s,a;\hat{\theta })-\tilde{Q}(s,a;{\theta }_i){)}^2].$$

$$d(s,a,r,s^{\prime })=|\tilde{Q}(s,a;\hat{\theta })-\tilde{Q}(s,a;{\theta }_i)|$$

$$p=(d-ϵ{)}^a,$$

$$P_k=\frac{p_k}{{\sum }_{j=1}^N{p}_j}$$

$$f({\delta }_i)=\{\begin{array}{ll}{e}^{-({\delta }_i-{\delta }_0)}& \text{ if }{l}_e=l_i\\ 0& \text{ otherwise}\end{array}$$

$$h(v)=(v-v_d{)}^2$$

$$a(v_t,v_{t-1})=(v_t-v_{t-1}{)}^2$$

$$g(l_t,l_{t-1})=\mathbb{I}(l_t\ne l_{t-1}).$$

$$r_t=-w_1\sum _{i=1}^{O_t}{f}_t({\delta }_i)-w_2{h}_t(v_t)-w_3\sum _{i=1}^{O_t}\mathbb{I}(f_t({\delta }_i)\ge 1-w_{4}a(v_t,v_{t-1})-w_{5}g(l_t,l_{t-1})$$

$${ϵ}_k=0.01+0.99e^{-\lambda k},$$

$$v_{l,t}\rho -v_{e,t}\rho +\frac{1}{2}{d}_{max}{\rho }^2>0.$$

$$\rho_{s}=\inf\Bigg{\{}\rho:\rho>\frac{2(v_{e,t}-v_{l,t})}{d_{max}}\Bigg{\}}.$$

Full text

K****eywords Autonomous vehicles  $\cdot$ Reinforcement learning  $\cdot$ Driving policy

1 Introduction

In recent years, there has been a growing interest in self-driving vehicles. Building such autonomous systems has been an active area of research [1, 2] for its high potential in leading to road networks that are much more safer and efficient. Although vehicle automation has already led to great achievements in supporting the driver in various monotonous and challenging tasks, see for example [3], rising the level of automation to fully-automated driving is an extremely challenging problem. This is mainly due to the complexity of real-world environments, including avoiding obstacles, and human driving behavior aspects.

According to Donges [4], autonomous driving tasks can be roughly classified into three categories; navigation, guidance, and stabilization. Navigation tasks are responsible for generating road-level routes. Tactical level guidance tasks are responsible for guiding autonomous vehicles along these routes in complex environments by generating tactical maneuver decisions. Finally, operational level stabilization tasks are responsible for translating tactical decisions into reference trajectories and then low-level controls that need to be tracked by the vehicle.

Several methodologies have been proposed for addressing the problem of generating efficient road-level routes. In the works of [5] and [6] the authors propose a navigation algorithm based on a route reservation mechanism, in order to generate road-level routes, and at the same time avoid traffic congestion. In [7] a computationally efficient algorithm that can scale to very large transportation networks is presented. The authors of [8] focus on "green" navigation by proposing a methodology to generate routes that minimize emissions. The work in [9] surveys exact algorithms for addressing the routing problem under capacity and travel time constraints. Finally, a comprehensive review regarding the generation of road-level routes in transportation networks can be found in [10]. Despite the research interest, navigation examined by the autonomous driving perspective can be considered as a mature technology, since already exist commercial and free applications for road-level route generation.

At the same time, vehicles are man-made products for which automotive industry has decades-long experience in vehicle dynamics modeling, see for example [11] and [12]. Therefore, the operational level stabilization tasks, also know as the acting part of autonomous driving, are well understood and modelled in control theory and robotics.

Tactical level guidance, referred also as driving policy, is crucial for enabling fully autonomous driving. On the contrary, however, to navigation and stabilization, tactical level guidance methodologies cannot be considered mature enough, in order to be applied to autonomous vehicles that move in unrestricted environments. A driving policy should be able to make decisions in real-time and in complex environments, in order to plan and update a vehicle path, which should be safe, collision-free, and user acceptable [13]. The requirement for real-time operation in highly complex environments, as well as the safety constraints make current driving policies inadequate for fully autonomous driving.

Based on the discussion so far, this work aims to contribute towards the development of a robust driving policy for autonomous vehicles that being capable of making decisions in real-time. The operation environment s restricted by considering vehicles that move on a highway. Highways consist a specific and very important type of transportation networks [14, 15]. Due to their high-capacity they can serve millions of people everyday, while at the same time, allow users to travel with higher speed and less accelerations/decelerations compared to urban transportation networks. The driving policy development problem is formulated from an autonomous vehicle perspective (ego vehicle), and, thus, there is no need to make any assumptions regarding the kind of other vehicles (manual driving or autonomous) that occupy the road.

Finally, the proposed methodology approaches the problem of driving policy development by exploiting recent advances in Reinforcement Learning (RL) combined with the responsibility sensitive safety model, proposed in [16]. The developed RL-based driving policy aims to avoid accidents (departures from the road and crashes with other vehicles), move the vehicle with a desired speed, minimize accelerations/decelerations, and minimize lane changes. The latter two criteria are also related to the comfort of vehicle passengers [17].

1.1 Related Work

The problem of path planning for autonomous vehicles can be seen as a trajectory generation problem corresponding to the creation of a quasi-continuous sequence of states that must be tracked by the vehicle over a specific time horizon. Trajectory generation has been widely studied in robotics [18]. Considering, however, road vehicles, path planning is a much more critical task, since passengers’ safety must be guaranteed.

Under certain assumptions, simplifications and conservative estimates, heuristic, hand-designed rules can be used for tactical decision making [19]. Such methods, however, are often tailored for specific non-complex environments and do not generalize robustly [20]. Therefore, they are not able to cope with the complexity of real-world environments and the diversity of driving conditions, let alone human driving behavior aspects. To overcome the limitations of rule-based methods, approaches based on the careful design and exploitation of potential field and optimal control methods have also been proposed.

Potential field methods generate a field, or, in other words, an objective function the minimization of which corresponds to the objectives of an agent. These methods are based on the design of potential functions for obstacles, road structures, traffic regulation and the goals to be achieved. Then, the overall objective function is expressed as the weighted sum of the designed potential functions. The minimization is achieved via the generation of a vehicle trajectory moving towards the descent direction of the overall objective function [21, 22, 23]. However, due to the fact that vehicle dynamics are not considered during decision making, the generated trajectory may turn out to be non-feasible to be tracked by the vehicle [24].

This drawback can be alleviated by formulating the trajectory generation problem as an optimal control problem, which inherently takes into consideration system dynamics. Specifically, optimal control approaches allow for the concurrent consideration of system dynamics and carefully designed potential fields [25]. In the work of [17] an optimal control methodology for vehicles’ trajectory planning in the context of cooperative merging on highways, is presented. The works of [26, 27] propose two optimal control based methodologies for trajectory planning, which incorporate constraints for obstacles, so as to keep the automated vehicle robustly far from them. In the same spirit, the works in [28] and in [29] design appropriate potential functions corresponding to the presence of obstacles, which, in turn, are incorporated in the objective function to generate a collision-free path. Optimal control approaches usually map the optimal control problem to a nonlinear programming (NLP) problem that can be solved using numerical NLP solvers, see for example [30, 31, 28]. Although, potential field and optimal control methods are quite popular due to the intuitive problem formulation [32], there are still open issues regarding the decision making process.

First of all, mapping the optimal control problem to a NLP problem and solving it by employing numerical NLP solvers, produces a locally optimal solution for which the guarantees of the globally optimal solution may not hold, and, thus, the safety guarantees for the generated trajectory may be compromised [33]. For this reason dynamic programming techniques have also been proposed for solving the optimal control problem. Although, dynamic programming techniques produce a globally optimal solution, due to the curse of dimensionality [34], they are restricted to small scale problems. Moreover, another problem faced with potential field and optimal control approaches is the strong dependency to a relatively simple environment model, usually with hand-crafted observation spaces, transition dynamics and measurements mechanisms. These assumptions limit the generality of these methods to complex scenarios, since they are not able to cope with environment uncertainties and measurements errors. Finally, optimal control methods are not able to generalize, i.e., to associate a state of the environment with a decision without solving an optimal control problem. This means that every time a sequence of decisions needs to be made an optimal control problem needs to be solved, even if exactly the same problem has been solved in the past. This requirement significantly increases the computational cost of these methods.

Due to its recent success, supervised deep learning has also been considered as an alternative approach for developing driving policies. In [35] a convolutional neural network is trained in a supervised manner to output continuous steering actions. In the work of [36] a recurrent neural network is trained to output a steering angle, after a driving intention has been estimated. The works in [37] and [38] also exploit end-to-end trainable neural networks that output feasible driving actions and affordance indicators (such as distance between cars). The aforementioned approaches are based on end-to-end trainable neural network architectures that are able to output low-level controls directly from input images. Therefore, this kind of driving policies correspond to the outcome of a supervised learning algorithm, where deep neural networks were trained to imitate the behavior of human drivers. However, such methods, first, result to black-box driving policies, which are susceptible to influence of drifted inputs, and second, are restricted to the limitations of end-to-end learning [39].

Very recently, RL methods have also been proposed as challenging alternative approaches towards the development of driving policies. RL-based approaches alleviate the strong dependency on hand-crafted simple environment models and dynamics, and, at the same time, can fully exploit the recent advances in deep supervised machine learning [40]. Along this line of research the work [41] utilizes a deep Q-Network to make decisions for intersection crossing, while the work [42] exploits a similar architecture to make decisions about lane changing in freeways. In [43], the authors propose a hierarchical RL-based approach for deriving a low-level driving policy capable of guiding a vehicle from an origin point to a destination point. In [44] a policy gradient RL approach is used to develop a driving policy for cooperative double merging scenarios. This approach combines a RL policy with a non-learnable mechanism to balance between efficiency and safety. Finally, the work in [45] presents some elements of efficient deep RL (empirically validated) for decreasing the learning time and increasing the efficiency of RL-based driving policies.

Despite the fact that only very recently reinforcement learning was employed for developing driving policies, experimental results appear very promising. The main drawback, however, of these approaches regards safety guarantees. Due to the fact that the probability of an accident is very small, learning based approaches, as shown in [16], cannot assure collision-free trajectories.

1.2 Proposed Work

This work proposes a RL-based approach towards the development of a driving policy for autonomous road vehicles. The proposed RL-based method has several advantages over potential field and optimal control methods. First of all, RL-based approaches are model-free. They make the assumption that there is a state-transition model that describes the system dynamics, which remains fixed. However, the exact form of this model is not required to be a priori known (typically such a model is considered unknown), but it is being inferred during training. Second, a driving policy based on RL is able to generalize. After training, a RL-based policy has inferred a mapping for associating a given state of the environment with a decision. In contrast to potential field and optimal control methods, whenever a decision needs to be made no problem needs to be solved; decision making can be done by simply evaluating the policy function. Third, since a RL-based driving policy has been estimated, it can be shared across multiple autonomous vehicles, which in turn can make decisions through the policy function evaluations. On the contrary, driving policy sharing is not possible when potential field and optimal control methods are used, since each vehicle needs to solve a decision making problem for its own sake. Finally, since no learning-based driving policy can guarantee absolute safety, our work is motivated by the formal responsibility sensitive safety model, proposed in [16], in order to derive and utilize ad-hoc rules that guarantee responsibility-wise safety. That is, the ad-hoc rules guarantee that the autonomous vehicles will not be responsible for any occurred accident. To the best of our knowledge, this work is one of the first attempts that try to derive a RL driving policy, combined with ad-hoc safety rules, targeting unrestricted highway environments, which are occupied by both autonomous and manual driving vehicles.

Furthermore, the proposed RL-based driving policy is compared against an optimal policy derived using dynamic programming, in terms of safety metrics, such as the number of collisions, and efficiency metrics, such as the average time the autonomous vehicle moves with the desired speed. Although, dynamic programming techniques, due to the curse of dimensionality [46], are restricted to small-scale problems, and are not suitable for real-time applications, they produce globally optimal solutions to an optimal control problem, i.e., optimal driving policies. Thus, the comparison of the proposed methodology against optimal driving policies, first, will result to an objective evaluation for the RL-based driving policy, and, second, can provide insights to the driving policy development problem.

The developed RL-based driving policy is also compared against manual driving using SUMO simulator. Through this comparison, the generalization ability and stability of the proposed RL-based driving policy to ensure reliability is evaluated; any learning system must generalize well to out-of-sample data, and be stable, i.e., small perturbations in the input should slightly affect the output. Specifically, the RL-based driving policy is applied to randomly generated driving scenarios (previously unseen driving conditions), with and without drivers’ imperfection and measurements errors. Drivers’ imperfection and measurements errors can be seen as disturbances, and can be incorporated into driving scenarios using appropriate settings in SUMO simulator.

Finally, preliminary results regarding the effect of autonomous vehicles on the overall traffic flow are provided. The RL-based driving policy, seen by an autonomous vehicle perspective, is a selfish policy. That is, each autonomous vehicle that follows the RL policy tries to achieve its own goals disregarding the rest of the vehicles. Such a behavior might have a negative effect on the overall traffic flow.

The rest of the paper is organized as follows: Section 2 describes the problem and the underlying assumptions, Section 3 gives a brief description of the RL framework, Section 3 presents in detail the development of the RL-based driving policy and Section 5 the derivation of ad-hoc rules towards the design of a collision-free trajectory. Section 6 presents the experimental setup and the experimental results, and Section 8 concludes this work.

2 Problem Description and Assumptions

The problem of path planning for an autonomous vehicle that moves on freeway, which is also occupied by manual driving vehicles is considered. Without loss of generality, it is assumed that the freeway consists of three lanes. The path planning algorithm, or in other, words the driving policy, should generate a collision-free trajectory for the autonomous vehicle to follow. Moreover, the generated trajectory should permit the autonomous vehicle to move forward with a desired speed, and, at the same time, minimize its longitudinal and lateral accelerations/decelerations. The aforementioned three criteria are the objectives of the driving policy, and therefore, the goal that the RL algorithm should achieve.

For the generation of an optimal trajectory using dynamic programming, the manual driving vehicles is required to move with a constant speed following the kinematics equations. The generation of the optimal trajectory, via dynamic programming, corresponds to the solution of a finite horizon optimal control problem. The aforementioned requirement assures that the dynamics of the system will be a priori and fully known, and no disturbances will be present in the system in order for the dynamic programming technique to produce the trajectory. However, for training the RL policy the aforementioned system dynamics are not given to the algorithm, and, thus, are considered unknown.

Regarding the SUMO simulator, the manual driving vehicles move on the freeway using the Krauss car following model [47]. It is assumed that letting the manual driving vehicles to move using the Krauss car following model will produce realistic driving behaviors. Moreover, manual driving vehicles should move forward with a desired speed. In order to generate realistic and customary traffic conditions, we assume that at least two categories of manual driving vehicles should be present at the freeway; manual driving vehicles that want to move faster than the autonomous vehicle, and manual driving vehicles that want to move slower. At this point, it should be stressed that, although the manual vehicles are moving using the Krauss model, this model is not given to the RL training algorithm, and, thus, from a RL point of view it is considered unknown.

During the trajectory generation this work does not assume any communication between the autonomous vehicle and other vehicles. Instead, the information available for the trajectory generation is obtained solely by sensors, such as cameras, LiDAR and proximity sensors, installed on the autonomous vehicle. This work also assumes the availability of a fusion module of the on-board sensors’ information, with the appropriate redundancy and cross-checking, to assure the usefulness and accuracy of the provided information. Using such sensors, the autonomous vehicle can estimate the position and the velocity of its surrounding vehicles. Therefore, the state representation of the autonomous vehicle and its surrounding environment, includes information that is associated solely with the position and the velocity of the vehicles present in the sensing area of the autonomous vehicle.

Furthermore, it is assumed that the freeway does not contain any turns. However, the generated vehicle trajectory essentially reflects the vehicle longitudinal position, speed, and its traveling lane. The derived trajectory needs to be tracked by the underlying vehicle control loops, based on high-definition maps. Therefore, for the trajectory specification, possible curvatures may be aligned to form an equivalent straight section [28].

Finally, the trajectory of the autonomous vehicle can be fully described by a sequence of goals that the vehicle should achieve. Each one of the goals should be achieved within a specific time interval, and represents vehicle’s desires, such as change lane, brake with a given deceleration, etc. These goals define the trajectory to be followed by the autonomous vehicle in a higher level, and cannot be directly used by the vehicle control loops. Instead, it is assume that the mechanism which translates these goals to low level controls and implements them is given.

Based on the aforementioned problem description and underlying assumptions, the main objective of this work is to develop a driving policy. The driving policy will exploit the information coming from a set of sensors installed on the autonomous vehicle, in order to set a goal for the vehicle to achieve, via a high-level action, during a specific time interval. In other words, the objective is to derive a function that will map the information about the autonomous vehicle, as well as, its surrounding environment to a specific goal and the corresponding high-level action for achieving it.

3 RL and Prioritized Experience Replay

In this work the development of a driving policy is being tackled as a RL problem, where the state-action value function $Q$ is approximated by a Double Deep Q-Network (DDQN) [48] using prioritized experience replay [49]. Therefore, for the sake of completeness, in this section the RL framework and the algorithm of prioritized experience replay are briefly presented.

3.1 Reinforcement Learning

In the RL framework, an agent interacts with the environment in a sequence of actions (selected by following a specific policy), observations, and rewards. In particular, at each time step $t$, the agent (in our case the autonomous vehicle) observes the state of the environment $s_{t}\in\mathcal{S}$ and, based on a specific policy, it selects an action $a_{t}\in\mathcal{A}$, where $\mathcal{S}$ is the state space and $\mathcal{A}=\{1,\cdots,K\}$ is the set of available actions. Then, the agent observes the new state of the environment, $s_{t+1}$, which is the consequence of applying the action $a_{t}$ at state $s_{t}$, and a scalar reward signal $r_{t}$, which is a quality measure of how good is to select action $a_{t}$ at state $s_{t}$.

The goal of the agent is to interact with the environment by selecting actions in a way that maximizes the cumulative future rewards, also known as future return. Future rewards are discounted by a factor $0\leq\gamma<1$ per time step, and the future return at time $t$ is defined as

[TABLE]

where parameter $T$ denotes how many time steps ahead $t$ are taken into consideration for calculating $R_{t}$. The non-negative discount factor $0\leq\gamma<1$ determines the importance of future rewards. In other words, it weighs future rewards, by giving higher weight to rewards received near than rewards received further in the future.

The interaction of the agent with the environment can be explicitly defined by a policy function $\pi:\mathcal{S}\rightarrow\mathcal{A}$ that maps states to actions. The maximum expected future reward achievable by following any policy after observing a state $s$ and selecting an action $a$ is represented by the optimal action-value function $Q^{*}(s,a)$, which is defined as

[TABLE]

The optimal action-value function obeys a very important identity known as Bellman equation. That is, if the optimal action-value function $Q^{*}(s_{t+1},a_{t+1})$ of the state $s_{t+1}$ at the next time step was known for all possible actions $a_{t+1}$, then the policy maximizing the future reward is to select the action maximizing the expected value of $r+\gamma Q^{*}(s_{t+1},a_{t+1})$, and, thus, the following

[TABLE]

holds for the optimal action-value function when state $s_{t}$ is observed and action $a_{t}$ is selected. The expectation in relation (3) is with respect to all possible states at the next time step.

The relation in (3) implies that the problem of estimating the optimal policy is equivalent to the estimation of $Q^{*}(s,a)$ for every pair $(s,a)\in\mathcal{S}\times\mathcal{A}$. Although, $Q^{*}(s,a)$ can be efficiently estimated when small scale problems need to be addressed [50], for large state spaces estimating $Q^{*}(s,a)$ for every possible $(s,a)$ pair is practically implausible. For such kind of problems, the optimal action-value function is approximated, $\tilde{Q}^{*}(s,a;\theta)\approx Q^{*}(s,a)$, using a learning machine, such as linear regression of neural networks [40], parameterized by $\theta$. Parameters $\theta$ are estimated by following an iterative procedure for minimizing a sequence of loss functions

[TABLE]

where $i$ stands for the iteration index. The $(s,a,r,s^{\prime})$ tuples used in relations (3) and (4) are generated by following an $\epsilon$-greedy policy that selects at a given state a greedy action with probability $1-\epsilon$ and a random action with probability $\epsilon$.

The aforementioned procedure for estimating $\theta$ looks like a regression problem in the supervised learning paradigm. However, there are two significant differences. First, the learning machine sets itself and follows the targets $\tilde{Q}(s,a;\theta_{i-1})$, which can lead to instabilities and divergence, and, second, the generated $(s,a,r,s^{\prime})$ tuples are not independently generated; a property that is required by many learning machines.

To overcome the first problem, two identical learning machines are used; one for setting the targets and one for following them. The machine that sets the targets is freezed in time, i.e., its parameters are fixed for several iterations. After a predefined number of iterations has passed, the parameters of the machine that sets the targets are updated by coping the parameters from the machine that follows the targets. If we denote as $\hat{\theta}$ the parameters of the machine that sets the targets, then the loss function $L_{i}(\theta_{i})$ in relation (4) is given by

[TABLE]

3.2 Prioritized Experience Replay Algorithm

To overcome the latter problem, a Prioritized Experience Replay (PER) algorithm is employed to break the correlations between the generated $(s,a,r,s^{\prime})$ tuples. The generated tuples are stored into a memory, and for minimizing (4) a training set $\mathcal{D}=\{(a,s,r,s^{\prime})_{j}\}_{j=1}^{n}$ is drawn from the memory according to a distribution that prefers tuples that do not fit well to the current estimate of the action-value function.

For estimating the sampling distribution, initially, the difference

[TABLE]

is computed for each tuple in memory and is updated after each iteration $i$. Then, the difference is converted to priority

[TABLE]

with $\epsilon>0$ to ensure that no tuple has zero probability of being drawn, and $0\leq a<1$ (when $a=0$ the uniform distribution over tuples is used). Finally, the priorities are translated into probabilities. In particular, a tuple $k$ has a probability

[TABLE]

of being drawn during the experience replay. Variable $N$ in (8) stands for the cardinality of memory.

4 Driving Policy

Having described the RL framework and the the prioritized experience replay algorithm, in this section, the RL-based approach utilized in this work is presented, along with the state and action representation, and the design of the scalar reward signal. Finally, the architecture of the employed neural network, and details about the implementation, as well as, the mechanism for generating $(s,a,r,s^{\prime})$ tuples for training the neural network are described.

4.1 State Representation

Autonomous vehicles are equipped with multiple sensors that enable them to capture heterogeneous and multimodal information about their surrounding environment. This allows for a wide variety of state representations. The selection, however, of the representation significantly affects the ability of an agent to learn. In this work, a state representation that, on the one hand, can be constructed using current sensing technologies, and, on the other, it allows the agent to efficiently learn is utilized.

Specifically, this work considers autonomous vehicles that move on a freeway with three lanes. It is assumed that the vehicle can sense the surrounding environment that spans 60 meters behind it and 100 meters ahead of it, as well as, its two adjacent lanes. This means that the autonomous vehicle can estimate the relative positions and velocities of other vehicles that are present in the aforementioned area. Note that with current LiDAR and camera sensing technologies such an assumption can be considered valid. A schematic representation of the sensed surrounding environment of a vehicle is presented in Figure 1(a).

In order to translate the information that can be sensed by the autonomous vehicle into a state vector, the sensed area is discretized into tiles of one meter length, see Fig. 1(b). In order for this discretization to be useful, the accuracy of the vehicle sensors must be in the order of centimeters, something that is feasible with current sensing technologies [51, 53, 54]. The value of the longitudinal velocity of the autonomous vehicle is assigned to the tiles beneath of it. To tiles occupied by other vehicles the value of their longitudinal velocity is assigned. The velocity of the other vehicles is estimated by using their positions in two subsequent time instances. The value of zero is given to all non occupied tiles that belong to the road and, finally, the value minus one to tiles outside of the road (the autonomous vehicle can sense an area outside of the road in case it occupies the left-most or the right-most lane).

Using the representation above, the sensed environment is transformed into a matrix with three rows and one hundred sixty columns. Moreover, this matrix contains information about the absolute velocities of vehicles, as well as, relative positions of other vehicles with respect to the autonomous vehicle. Finally, the vectorized version of this matrix, that is a vector with 480 elements, is used to represent the state of the environment at each specific time step.

The proposed state representation can be easily obtained by sensors installed on an autonomous vehicle. Despite the fact that this representation is relatively simple, as it can be seen in Section 6, it contains adequate information for obtaining robust driving policies. More realistic and informative state representations can be constructed. For example current pattern and object recognition methods can be utilized to classify vehicles and, thus, incorporate into the state representation information regarding the type of surrounding vehicles and their size. In addition, if we assume that vehicles are equipped with communication enabling technologies, then vehicle-to-vehicle communication can be used to enhance the state representation with information regarding vehicles’ longitudinal and lateral accelerations, while vehicle-to-infrastructure communication can provide information regarding the state of the network. Although, more accurate state representations can be constructed, using a simple state representation, like the proposed one, permits to gain insights with respect to the behavior of the derived policy. Moreover, we deliberately do not assume any communication between the vehicles, to make the training of the RL policy much harder, and, at the same time, be able to evaluate its behavior under minimal assumptions. Finally, this work is based on the argument that RL based techniques can be proved very valuable towards the developments of driving policies, even in mixed driving scenarios, and thus, it can be seen as a preliminary proof-of-concept.

4.2 Action Representation

Seven available actions are defined; i) change lane to the left, ii) change lane to the right, iii) accelerate with a constant acceleration of $1m/s^{2}$ or $2m/s^{2}$, iv) decelerate with a constant deceleration of $-1m/s^{2}$ or $-2m/s^{2}$, and v) move with the current speed at the current lane, see Table 1. For the acceleration and deceleration actions feasible acceleration and deceleration values are used to ensure that the autonomous vehicle will be able to implement them. Moreover, the autonomous vehicle is making decisions by selecting one action every one second, which implies that the first two actions are also feasible, that is, a moving car is able to change lane in a time interval of one second.

Using the aforementioned action representation, each action can be seen as a goal or desire of the autonomous vehicle that should be achieved during one second. Practically, the first six actions represent goals that are associated with the avoidance of obstacles. The third to sixth actions represent also goals that are related to the fact that the autonomous vehicle should move forward with a desired speed. Finally, the seventh action implies that the vehicle is moving with the desired speed and there are no obstacles to avoid.

Note that the goal of this work is to develop a driving policy by approximating through RL the action-values $Q(s,a)$ for every possible $(s,a)\in\mathcal{S}\times\mathcal{A}$ pair. Therefore, adopting an action space with small cardinality can significantly simplify the problem leading to faster training. Moreover, the authors of [45] argue that low-level control tasks can be less effective and/or robust for high-level driving policies. For these reasons an action space like the one presented above is used, instead of lower level commands such as longitudinal and lateral accelerations.

Finally, by using an action set of goals, the RL-based driving policy makes high-level decisions for leading the autonomous vehicle to a desired state. The implementation of these goals can efficiently take place by exploiting a separate non-learnable module, such as dynamic programming. This low-level module will produce state trajectories by translating each specific desire to lower level commands, such as longitudinal and lateral accelerations. These state trajectories may then be used as a reference by the vehicle throttle and brake controllers, which are designed on the basis of vehicle dynamics, to produce the actual vehicle movement on the road. As mentioned in Section 2, the development of such a module is beyond the scope of this work, and, thus, it is assumed that is given.

4.3 Reward Signal Design

The reward signal is a measure of the quality of a selected action at a specific state, and is the only mean through which a policy can be evaluated. So, designing appropriate rewards signals is the most important tool for shaping the driving behavior of an autonomous vehicle.

For driving scenarios, the autonomous vehicle should be able to avoid collisions, move with a specific desired speed and avoid unnecessary lane changes and accelerations. Therefore, the reward signal should reflect all these objectives by employing one penalty function for collision avoidance, one that penalizes deviations from the desired speed and two penalty functions for unnecessary lane changes and accelerations/decelerations.

The penalty function for collision avoidance should feature high values at the gross obstacle space, so that the autonomous vehicle is repulsed, and potentially unsafe decisions are suppressed; and low (or virtually vanishing) values outside that space. To this end, the exponential penalty function

[TABLE]

is adopted. In (9) $\delta_{i}$ is the longitudinal distance between the automated vehicle and the $i$-th obstacle (the $i$-th vehicle in its surrounding environment), $\delta_{0}$ stands for the minimum safe distance, and, $l_{e}$ and $l_{i}$ denote the lanes occupied by the autonomous vehicle and the $i$-th obstacle, respectively. Note that this function is activated only when the automated vehicle and an obstacle are at the same lane. Finally, if the value of (9) becomes greater or equal to one, then the driving situation is considered very dangerous and it is treated as a collision.

The vehicle mission is to advance with a longitudinal speed close to a desired one. Thus, the quadratic term

[TABLE]

that penalizes the deviation between the vehicle speed and its desired speed, is incorporated in the reward. In (10) the variable $v$ stands for the longitudinal speed of the autonomous vehicle, while the constant $v_{d}$ represents its desired longitudinal speed.

Two terms are also introduced; one for penalizing accelerations/decelerations, and one for penalizing unnecessary lane changes. For penalizing accelerations the term

[TABLE]

is used, while for penalizing lane changes the term

[TABLE]

is used. Variables $v_{t}$ and $l_{t}$ correspond to the speed and lane of the autonomous vehicle at time step $t$, while $\mathbb{I}(\cdot$) is the indicator function.

The total reward at time step $t$ is the negative weighted sum of the aforementioned penalty terms, that is

[TABLE]

In (13) the third term penalizes collisions and variable $O_{t}$ corresponds to the total number of obstacles that can be sensed by the autonomous vehicle at time step $t$. The selection of weights defines the importance of each penalty function to the overall reward. In this work the weights were set, using a trial and error procedure, as follows: $w_{1}=1$, $w_{2}=0.5$, $w_{3}=20$, $w_{4}=0.01$, $w_{5}=0.01$. The largest weighting factors are associated with the terms that penalize collisions and model obstacle avoidance, since the derived policy should generate collision free trajectories. The weighting term associated with the desired speed of the vehicle defines how aggressive and/or how conservative will be the derived driving policy. Using a small value for this weight will result to a conservative policy that will advance the vehicle with very low speed or, even worse, keep the vehicle immobilized by setting its speed equal to zero. Finally, the values of the weighting factors associated with lane changes accelerations/decelerations are small in order to enable the vehicle to make maneuvers, such as overtaking other vehicles.

4.4 Neural Network Architecture

As mentioned before, the goal of this work is to develop a driving policy by approximating through RL the action-values $Q(s,a)$ for every possible $(s,a)\in\mathcal{S}\times\mathcal{A}$ pair. Towards this direction a fully connected feed forward neural network is utilized, due to its universal function approximation property [52].

Specifically, the action-values $Q(s,a)$ for each pair $(s,a)\in\mathcal{S}\times\mathcal{A}$ are approximated by using a neural network that maps a specific state $s\in\mathcal{S}$ to the action-values $Q(s,a_{s,i})$, where $\{a_{s,i}\}_{i}$ is a non empty set that contains all actions that can be selected by the policy when the agent is at state $s$. In this work, the DDQN approach is followed which utilizes two identical neural networks with two hidden layers, consist of 256 and 128 neurons respectively. The first neural network is responsible for setting the targets, while the second one is responsible for following them. The synchronization between the two neural networks is realized every 1000 epochs. For more information regarding the DDQN model please refer to [48].

4.5 Training Set Generation and Policy Training

For generating $(s_{t},a_{t},r_{t},s_{t+1})$ tuples that will be used for training the DDQN, two different microscopic traffic flow simulators are used. The first one is a custom made simulator that moves the manual driving vehicles with constant speed using the kinematics equations. The second simulator is the established SUMO111www.sumo.dlr.de/ microscopic traffic flow simulator. By exploiting traffic flow simulators driving scenarios can be simulated. For each one of the simulation steps during a simulated scenario, following the approach described in Section 4, one $(s_{t},a_{t},r_{t},s_{t+1})$ tuple can be generated using information coming directly from the simulator.

After the collection of a set of $(s_{t},a_{t},r_{t},s_{t+1})$ tuples the training of the RL policy is starting following the procedures described in Section 3. It should be mentioned that during policy training (and testing) we implemented a rule-based action masking [45] for changing lanes. Our choice is justified by the fact that in some driving situations, undesirable lane changes can be straightforward identified, e.g. lane changes that result to immediate collisions. In such cases undesirable lane changes are filtered out, instead of letting the agent to learn to avoid that actions. The benefits from action masking is twofold. First, it restricts the actions space, and, thus, it speeds up the learning process. Second, selection of inferior actions caused by the variance in observation will be avoided resulting to a policy that is less prone to false positives and easier to debug. Besides the aforementioned action masking, during training, no other safety mechanisms are applied on the behavior of the autonomous vehicle. On the contrary, regarding the manual driving cars, all safety mechanisms are enabled. Therefore, in case of a collision we are sure that the vehicle that caused the collision is the autonomous one.

These are the general rules applied during the driving scenarios generation (for training and testing the RL-based driving policy) using both of the aforementioned microscopic traffic-flow simulators. Depending on the specific characteristics of each experiment extra rules may be applied. These are described in the corresponding subsections of Section 6.

4.6 Implementation details

For training the network we set the discount factor $\gamma=0.995$ [see relation (1)], we used a memory of 2000 samples capacity, a mini-batch of 64 samples and the ADAM optimizer with learning rate $0.003$, $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The exploration factor $\epsilon_{t}$ at each step is annealed by

[TABLE]

where $k$ stands for the index of the latest training step and $\lambda$ was set equal to $7.5\cdot 10^{-6}$. Finally, the training process started with $\epsilon_{1}=1.0$ and terminated when $\epsilon_{k}=0.01$.

5 Safety Rules

As mentioned before, no learning based driving policy can guarantee a collision free trajectory. There will always be corner cases (very rare events) that the learning algorithm will not encounter during its training phase. Therefore, it cannot be assured that the decisions corresponding to such event will be correct [for a formal proof of this result see [16] Lemma 2]. Moreover, a vehicle might be involved in an accident without being responsible for it. For these reasons the authors of [16] derive ad-hoc rules to guarantee responsibility-sensitive safety, that is, to guarantee that an autonomous vehicle will never cause an accident, even if it will be involved in one.

The derivation of safety rules in this work is motivated by the responsibility-sensitive framework. There is, however, a main difference between the setting in [16] and our setting. The authors in [16] assume that the road is occupied only by autonomous vehicles whose behaviour can be programmed. In our case, there is no such assumption. On the contrary, mixed driving scenarios are considered, where the road is occupied both by autonomous and manual driving vehicles. This implies that the behaviour of manual driving vehicles cannot be affected neither programmed. By restricting attention on vehicles that move on a highway the aforementioned assumption can be removed. This allows to assume that extreme events, such as vehicles that stop suddenly, will not occur.

Restricting attention on highways, permits also the simplification of the responsibility-safety framework by considering two types of collisions. An autonomous vehicle can cause an accident, firstly, if it moves faster than its leader and violates a minimum time gap, and, secondly, during lane changes. In the following we derive rules for avoiding these two types of collisions. Please note, that the information that can be used to derive such rules is only the information available to the autonomous vehicle, that is the positions and the velocities of the vehicles surrounding it.

In order to avoid the first type of collisions, the minimum safety time gap $\rho_{s}$ that must be maintained between the autonomous vehicles and its leader needs to be estimated. Obviously, the minimum safety time gap makes sense only when the autonomous vehicle is moving faster that its leader. Let us denote as $v_{e,t}$ and as $v_{l,t}$ the longitudinal speeds of the autonomous vehicle and its leader vehicle, respectively. Also, let us denote as $d_{max}$ the maximum feasible deceleration of the autonomous vehicle. In order to avoid the first type of collisions after a time interval $\rho$, the following inequality should hold:

[TABLE]

Solving for $\rho$ the minimum safety time gap $\rho_{s}$ can be obtained by

[TABLE]

Based on relation (16), the autonomous vehicle before performing an action, different than lane change actions, evaluates the minimum safety gap with respect to its leader. If the minimum time gap is violated, the autonomous vehicles decelerates with $d_{max}$ until its speed becomes equal to the speed of its leader. Otherwise, it performs the RL selected action.

Regarding the second type of collisions that can be caused by lane changes, two different cases should be considered. The autonomous vehicle should avoid collisions with its leader vehicle and with its follower vehicle in the newly selected lane. In the first case, estimates the minimum safety time gap $\rho_{s}$ with respect to its leader in the newly selected lane. If the minimum time gap is not violated the RL lane change action is performed. Otherwise, the autonomous vehicle selects the last action of the action set $\mathcal{A}$ [see Section 4.2], that is to retain current lane and move with current speed, and checks for the first type of collisions. In order to avoid the collisions with its follower vehicle in the newly selected lane, the autonomous vehicle is not permitted to change lane if the follower vehicle moves faster. In this case again, the autonomous vehicle selects the last action of the action set $\mathcal{A}$, and checks for the first type of collisions. The rule for avoiding collisions between the autonomous vehicle and its follower is very conservative. However, since the RL-based driving policy cannot affect the behavior of the follower, and at the same time has no access to its maximum feasible deceleration (in order to relax this rule by estimating a safety time gap), such a rule is the only way to guarantee no collisions of the second type.

Although, the derived safety rules lead to a more conservative driving policy, as it can be seen in the experimental validation of the proposed approach, they permit the autonomous vehicle to advance with its desired speed and at the same time avoid collisions.

6 Experiments

In this work three different sets of experiments were conducted. In the first set of experiments a simplified microscopic traffic flow simulator is utilized in order to compare the behavior of the RL-based driving policy against an optimal policy derived via Dynamic Programming. In the second set of experiments the established microscopic traffic simulator SUMO is used. Three different types of experiments are conducted. First, the behavior of the autonomous vehicle is evaluated when it is controlled by the derived RL-based policy and when it is controlled by SUMO. Second, the robustness of the derived policy with respect to measurement errors is evaluated. Finally, in the third set of experiments, the effect of vehicles that move following the RL-based policy on traffic flow is investigated. In the following the details of the experimental setup and the obtained results are presented.

6.1 RL-based driving policy and Dynamic Programming

Dynamic Programming techniques can produce optimal policies assuming that no disturbances occur in the system. Due to this fact, for this set of experiments, a simplified custom made microscopic traffic simulator was developed and utilized. This simulator moves the manual driving vehicles with constant longitudinal velocity using the kinematics equations. Moreover, the manual driving vehicles are not allowed to change lanes. Despite its simplifying setting, this set of experiments allow the comparison of the RL driving policy against an optimal policy derived via Dynamic Programming. At this point it should be mentioned that for this set of experiments the ad-hoc safety rules derived in Section 5 are disabled, in order to gain insights regarding the safety aspects of the RL-based driving policy.

For training the DDQN, driving scenarios of 60 seconds length were generated. In these scenarios one vehicle enters the road every two seconds, while the tenth vehicle that enters the road is the autonomous one. All vehicles enter the road at a random lane, and their initial longitudinal velocity is randomly selected from a uniform distribution ranging from $12m/s$ to $17m/s$. Finally, the desired speed of the autonomous vehicle is set equal to $21m/s$.

The RL driving policy is compared against an optimal policy derived via Dynamic Programming under four different road density values. For each one of the different densities 100 scenarios of 60 seconds length were simulated. In these scenarios, the simulator moves the manual driving vehicles, while the autonomous vehicle moves by following the RL policy and by solving a Dynamic Programming problem with 60 seconds horizon (which utilizes the same objective functions and actions as the RL algorithm). Finally, statistics regarding the number of collisions and lane changes, and the percentage of time that the autonomous vehicle moves with its desired speed for both the RL and Dynamic Programming policies are extracted. At this point it has to be mentioned that Dynamic Programming is not able to produce the solution in real time, and it is just used for benchmarking and comparison purposes. On the contrary the RL policy, at a given state can select an action very fast, since this selection corresponds to one evaluation of the neural network function at the corresponding state.

Table 2 summarizes the results of this comparison. The four different densities are determined by the rate at which the vehicles enter the road, that is, 1 vehicle enters the road every 8, 4, 2, and 1 seconds. The RL policy is able to generate collision free trajectories, when the density is less than or equal to the density used to train the network. For larger densities, however, the RL policy produced 2 collisions every 100 scenarios. In terms of efficiency, the optimal Dynamic Programming policy is able to perform more lane changes and advance the vehicle faster.

6.2 RL-based driving policy and SUMO policy

In this set of experiments the behavior of the autonomous vehicle when it follows the RL policy and when it is controlled by SUMO is evaluated. The training of the RL policy took place using scenarios generated by the SUMO simulator. During the generation of scenarios, all SUMO safety mechanisms are enabled for the manual driving vehicles and disabled for the autonomous vehicle. Furthermore, the manual driving cars is not permitted to implement cooperative and strategic lane changes. Such a configuration for the lane changing behavior, impels the autonomous vehicle to implement maneuvers in order to achieve its objectives. Moreover, in order to simulate realistic scenarios two different types of manual driving vehicles are used; vehicles that want to advance faster than the autonomous vehicle and vehicles that want to advance slower. Finally, the density was equal to 600 veh/lane/hour. For the evaluation of the trained RL policy, different driving scenarios, described in the following subsections, were simulated.

6.2.1 Evaluation of the derived RL driving policy and safety rules

In this set of experiments different driving scenarios were simulated; i) 100 driving scenarios during which the autonomous vehicle follows the RL driving policy without the ad-hoc safety rules derived in Section 5, ii) 100 driving scenarios during which the autonomous vehicle follows the RL driving policy with the ad-hoc safety rules, iii) 100 driving scenarios during which the default configuration of SUMO was used to move forward the autonomous vehicle (cooperative and strategic lane changes are enabled for the autonomous vehicle), and iv) 100 scenarios during which the behavior of the autonomous vehicle is the same as the manual driving vehicles, i.e. it does not perform strategic and cooperative lane changes. The duration of all simulated scenarios was 60 seconds. The aforementioned scenarios’ generation framework was applied for two different driving conditions. In the first one the desired speed for the slow manual driving vehicles was set to $18m/s$, while in the second one to $16m/s$. For both driving conditions the desired speed for the fast manual driving vehicles was set to $25m/s$. Furthermore, in order to investigate how the presence of uncertainties affects the behavior of the autonomous vehicle, simulated scenarios where drivers’ imperfection was introduced by appropriately setting the $\sigma$ parameter in SUMO ($0\leq\sigma\leq 1$ with $\sigma=0$ to imply a perfect driver) were also used. Finally, the behavior of the autonomous vehicles was evaluated in terms of i) collision rate, and ii) average speed per scenario.

Table 3 summarizes the results of this comparison when the ad-hoc safety rules are disabled. In this way the safety levels of the RL-based driving policy can be experimentally quantified. In Table 3, SUMO default corresponds to the default SUMO configuration for moving forward the autonomous vehicle, while SUMO manual to the case where the behavior of the autonomous vehicle is the same as the manual driving vehicles. Irrespective of whether a perfect ($\sigma=0$) or an imperfect ($\sigma=0.5$) driver is considered for the manual driving vehicles, the RL policy is able to move forward the autonomous vehicle faster than the SUMO simulator, especially when slow vehicles are much slower than the autonomous one. However, it results to a collision rate of 2%-4%, which is its main drawback. No guarantees for collision-free trajectory is the price paid for deriving a learning based approach capable of generalizing to unknown driving situations and inferring driving actions with minimal computational cost.

However, when the ad-hoc safety rules are enabled the derived RL driving policy achieves to provide collision free trajectories. The average speed of the autonomous vehicle slightly decreases after the application of ad-hoc rules, but again the derived policy advances the autonomous vehicle faster than the SUMO policies. Specifically, when the speed of the slow vehicles is $18m/s$ the RL-based policy with the ad-hoc safety rules advances the autonomous vehicle 2% and 2.6% faster than the SUMO default policy for $\sigma=0.0$ and $\sigma=0.5$ respectively. For the case where the speed of the slow vehicles is $16m/s$, the improvement, in terms of speed, of the RL-based policy over the SUMO default policy is more significant. In particular, the RL-based policy advances the autonomous vehicle 8% and 12% faster than the SUMO default policy for $\sigma=0.0$ and $\sigma=0.5$ respectively.

The aforementioned results suggest that the RL-based driving policy is not significantly more efficient than the SUMO default policy when the average speed of the manual driving vehicles is close to the desired speed of the autonomous vehicle. However, when the deviation between the desired speed of the autonomous vehicle and the average speed of the manual driving vehicles increases, the RL-based driving policy is able to advance the autonomous vehicle much faster.

6.2.2 Evaluation of the derived RL driving policy under measurement errors

In this set of experiments the robustness of the RL-based driving policy, with the application of ad-hoc safety rules, is evaluated with respect to measurement errors regarding the position of the manual driving vehicles. At each time step, measurement errors proportional to the distance between the autonomous vehicle and the manual driving vehicles are introduced. Two different error magnitudes were used; $\pm 5\%$ and $\pm 10\%$. The RL policy was evaluated in terms of collisions and average speed in 100 driving scenarios of 60 seconds length for each error magnitude. In these scenarios the desired speed of the slow vehicles is $16m/s$. Finally, for these experiments, perfect and imperfect drivers were also considered.

The results of this evaluation are presented in Table 4. Despite the introduction of noise, the RL-based driving policy is able to produce collision-free trajectories, and at the same time, retain a high speed for the autonomous vehicle. In particular, the introduction of 5% noise does not seem to affect the average speed of the vehicle. By increasing the noise to 10% the average speed of the vehicle slightly decreases compared to the case with noiseless measurements. Fig. 2 presents the speed trajectories of the autonomous vehicle when different drivers’ imperfection and different magnitude of noises are introduced. The solid green line represents the mean speed of the vehicle over all 100 scenarios, while the shaded area represents 1 standard deviation of the speeds below and above their mean value. Irrespective of the introduced uncertainties, during the first steps of the simulation the autonomous vehicle increases its speed to reach a speed close to its desired one, and then, it retains this speed. Moreover, by increasing the noise, the deviation of the speeds over the 100 scenarios increases. This, however, is a rational behavior, since increasing the uncertainty, in terms of noisy measurements, will increase the variance during the decision making process.

6.2.3 Evaluation of the derived RL driving policy with unknown vehicle types

In this set of experiments the robustness of the derived RL-based driving policy is evaluated when the road is occupied by types of vehicles that were not present during the training phase. The RL-based driving policy was trained using driving scenarios where the road was occupied by passenger manually driving vehicles that were moving faster and slower than the autonomous vehicle. In this set of scenarios the road is occupied by the previously mentioned passenger vehicles, but also by truck, buses and motorcycles. The percentage of these types of vehicles as well as their desired speed are presented in Table 5. Under this experimental setting the robustness of the derived driving policy can be evaluated when vehicles of different sizes and different desired speeds occupy the road. Towards this direction 100 driving scenarios considering perfect drivers and 100 scenarios considering drivers’ imperfections were simulated. All driving scenarios were 60 seconds long. Finally, the RL-based driving policy was evaluated in terms of collisions and average speed with which the autonomous vehicle moves forward.

Table 6 presents the RL driving policy evaluation results for the aforementioned set of experiments. By comparing these results with the results in Table 3, it can be seen that the average speed of the autonomous vehicle is slightly decreases by 0.13 m/s and 0.14 m/s, for $\sigma=0.0$ and $\sigma=0.5$ respectively, when types of vehicles not seen during the training phase are present in the road. This decrease is mainly due to the randomness during driving scenarios generation and not due to the presence of trucks, buses and motorcycles on the road. More importantly, the RL driving policy is able to produce collision free trajectories despite the fact that the road is occupied by types of vehicles not seen during the training phase. This is justified by two facts. First, the proposed state representation utilizes encodes about the position and the velocity of manual driving vehicles present on the road. This kind of information can be obtained and encoded for any vehicle irrespective of it type. Second, the development and application of the proposed safety rules compensates for the presence of manually driving vehicles of different sizes. It should be mentioned, however, that more realistic and accurate state representations (see Section LABEL:ssec:state) can also be utilised to explicitly encode vehicles size information in state representation.

6.2.4 Evaluation of the derived RL driving policy under rainy weather conditions

In this set of experiments the RL driving policy is evaluated under rainy weather driving conditions. Rainy weather shifts the fundamental diagram to the left, which implies, on the one hand, that the vehicles move slower, and on the other, that their maximum acceleration/deceleration becomes lower. In order to simulate rainy weather driving scenarios, the desired speed of all vehicles, except the autonomous one, was decreased by 10%, and their maximum acceleration/deceleration by 30%. Moreover, drivers’ imperfections were also included by setting $\sigma=0.5$. Finally, different types of vehicles, that is slow and fast passenger vehicles, truck, buses and motorcycles, were also present on the road. 100 driving scenarios of 60 seconds long were simulated, and the derived driving policy was evaluated in terms of number of collisions and average speed with which the autonomous vehicle moves forward. Table 7 presents the result of this evaluation. The RL-based driving policy is able to produce collision free trajectories and, at the same time, move forward the autonomous vehicle with a speed larger than 19 m/s. The longitudinal velocity of the autonomous vehicle is slightly lower (0.19 m/s) than the previous experiments. This, however, is mainly caused by the decrease in overall traffic flow due to weather conditions.

6.3 The effect of the RL-based driving policy on traffic flow

In this set of experiments preliminary results on the effect of autonomous vehicles on the overall traffic flow are presented. Four different experiments are conducted by varying the percentage of autonomous vehicles that occupy the road. In the first experiment all vehicles are manual driving, that is the percentage of autonomous vehicles is zero. For the rest three experiments percentages of 5%, 10% and 15%, respectively, are used. For each experiment 100 scenarios of 120 seconds length were simulated and for each scenario the average speed of all vehicles on the road is computed, which is an indicator of the flow; the higher the average speed the higher is the flow. For all experiments and all scenarios the desired speed of manual driving vehicles is $16m/s$ and the option for cooperative and strategic maneuvers is disabled, while the desired speed for all the autonomous vehicles is $21m/s$. In this way the behavior of the manual driving vehicles can be seen as a moving bottleneck.

The results of these experiments are presented in Table 8. For each one of the experiments the average speed is reported. As a baseline the case where the percentage of autonomous vehicles is zero is considered, and the relative improvement of the rest of the cases (5%, 10%, and 15% autonomous vehicles) against this one is also reported. The average speed for the baseline is $15.32m/s$, a little bit lower than the desired speed of the manual driving vehicles. This happens because the manual driving vehicles should satisfy the safety constraints imposed by SUMO. When the percentage of autonomous vehicles increases to 5% the average speed of the vehicles is $15.41m/s$ resulting in a very small improvement of $0.6\%$ over the baseline. In this case the autonomous vehicles move faster than the manual driving ones. Their percentage, however, is very small, and, thus, they only slightly improve the average speed compared to the baseline. Increasing more the percentage of autonomous vehicles to 10% results to an average speed of $16.11m/s$ and $5.1\%$ improvement compared to the baseline. Increasing, however, more the percentage of autonomous vehicles to 20% results to an improvement of $1.3\%$ over the baseline, which is smaller than the improvement of the previous case. This mainly happens due to the selfish behavior of the autonomous vehicles.

Autonomous vehicles want to move faster than the manual driving cars, and in order to achieve that they have to perform maneuvers. Keeping the density of autonomous vehicles low permits the performance of maneuvers, and thus, the faster advancement of the autonomous vehicles. Increasing, however, the density of autonomous vehicle above a threshold, makes the performance of maneuvers more difficult, since each one autonomous vehicle, present in a limited space, wants to perform its own maneuvers in a selfish way. This results in competitive behaviors among the autonomous vehicles, which has a negative effect on the overall traffic flow.

These preliminary results show that selfish and competitive behaviors deteriorate the overall traffic flow. Deriving an RL-based driving policy trained on scenarios where the manually driven vehicles occupy a selfish behavior will not improve the overall traffic flow. Due to limited space and the large number of maneuvers performed by the manually driven vehicles the RL training algorithm will result to a very conservative policy. In our view, the only way to improve the overall traffic flow, under mixed driving scenarios, is to derive cooperative driving policies for clusters of autonomous vehicles, in order to achieve not vehicle-centric, but overall traffic flow goals. This could be done by introducing appropriate penalty terms regarding the overall traffic flow, such as minimum travel time or average traffic flow, in the reward function. Deriving, however, cooperative RL-based driving policies for clusters of autonomous vehicles is outside the scope of this work.

7 Discussion

The simulation results presented in Section 6 indicate that the derived RL-based driving policy is more efficient, for moving forward the autonomous vehicle, than the car following model used by SUMO simulator. At the same time, the derived policy can produce collision free trajectories, and it seems to be robust under measurement errors, different types of vehicles and weather conditions. Although, the current work makes the first steps towards the exploitation of deep RL techniques for autonomous vehicles’ path planning, the proposed methodology is not yet ready for real-world adoption. More complicated scenarios should be generated and utilised during the training and testing phases, such as scenarios where the autonomous vehicle is approaching a crash site ahead, heavy traffics, highway merging, emergency lane switching and night driving.

Being able to identify the limitations of the current work motivates our ongoing and future work, which comprises i) training and testing an RL-based driving policy under more complicated and realistic scenarios, ii) derive more accurate state representations by exploiting vehicle-to-vehicle and vehicle-to-infrastructure communication technologies, and iii) move from a selfish driving policy to the derivation of a cooperative driving policy in order to achieve not vehicle-centric, but overall traffic flow goals.

8 Conclusions

In this work, the problem of path planning for an autonomous vehicle that moves on a freeway is considered. For addressing this problem RL techniques are employed to derive a driving policy. The driving policy is implemented using a DDQN. Two different simulators to train and validate the derived driving policy are used; a custom made microscopic traffic flow simulator and the established SUMO microscopic traffic flow simulator.

The custom made microscopic traffic flow simulator is utilized for comparing the RL-based driving policy against an optimal policy derived via Dynamic Programming. The results of this comparison indicated that, although, Dynamic Programming can advance the autonomous vehicle faster than the RL-based driving policy, it cannot produce the trajectory in real time. Moreover, Dynamic Programming requires a priori and exact knowledge of the system dynamics in a disturbance free environment to produce an optimal solution. Due to these facts, an RL-based driving policy that incorporates the ad-hoc safety rules [see Section 5] can be proved a valuable approach for emerging driving behaviors with very low computational cost, minimal or no assumptions about the environment, and the capability to generalize to driving situations that are not known a priori.

The SUMO simulator is utilized in order to train and validate the RL-based driving policy under customary and realistic traffic scenarios. Since, no learning based approach can guarantee collision-free trajectories, ad-hoc safety rules are derived motivated by the responsibility-safety framework presented in [16]. The derived RL-based driving policy is compared against SUMO policies with and without the introduction of uncertainties. The results of this comparison indicated that the autonomous vehicle following the RL-based policy is able to achieve higher scores.

Finally, preliminary results regarding the effect of selfish autonomous vehicles behavior on the overall traffic flow are presented. These results suggest that, although, an individual autonomous vehicle that follows a selfish policy can achieve its goals, when multiple autonomous vehicles follow a selfish policy their impact on the overall traffic flow is negative. Selfish policies lead to competitive behaviors that deteriorate the overall traffic flow. This effect is known as the user optimum versus system optimum trade-off.

9 Acknowledgement

This research is implemented through and has been financed by the Operational Program ”Human Resources Development, Education and Lifelong Learning” and is co-financed by the European Union (European Social Fund) and Greek national funds.