CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms

TL;DR

CoRide introduces a hierarchical multi-agent reinforcement learning framework for joint order dispatching and fleet management in ride-hailing platforms, optimizing city-wide operations and improving revenue and user experience.

Contribution

The paper proposes a novel hierarchical multi-agent reinforcement learning framework, CoRide, for joint decision-making in ride-hailing, addressing multi-scale challenges with a unified ranking-based action formulation.

Findings

Outperforms strong baselines in real-world city data

Enhances platform revenue and user experience

Effectively manages multi-scale ride-hailing operations

Abstract

How to optimally dispatch orders to vehicles and how to tradeoff between immediate and future returns are fundamental questions for a typical ride-hailing platform. We model ride-hailing as a large-scale parallel ranking problem and study the joint decision-making task of order dispatching and fleet management in online ride-hailing platforms. This task brings unique challenges in the following four aspects. First, to facilitate a huge number of vehicles to act and learn efficiently and robustly, we treat each region cell as an agent and build a multi-agent reinforcement learning framework. Second, to coordinate the agents from different regions to achieve long-term benefits, we leverage the geographical hierarchy of the region grids to perform hierarchical reinforcement learning. Third, to deal with the heterogeneous and variant action space for joint order dispatching and fleet management, we design the action as the ranking weight vector to rank and select the specific order or the fleet management destination in a unified formulation. Fourth, to achieve the multi-scale ride-hailing platform, we conduct the decision-making process in a hierarchical way where a multi-head attention mechanism is utilized to incorporate the impacts of neighbor agents and capture the key agent in each scale. The whole novel framework is named as CoRide. Extensive experiments based on multiple cities real-world data as well as analytic synthetic data demonstrate that CoRide provides superior performance in terms of platform revenue and user experience in the task of city-wide hybrid order dispatching and fleet management over strong baselines.

Tables

Table 1. Table 1. Performance comparison of competing methods in terms of ADI and ORR with respect to the performance of RAN. For a fair comparison, the random seeds that control the dynamics of the environment are set to be the same across all methods.

City	City A		City B		City C
Metrics	Normalized ADI	Normalized ORR	Normalized ADI	Normalized ORR	Normalized ADI	Normalized ORR
DQN	+5.71% $\pm$ 0.02%	+2.67% $\pm$ 0.01%	+6.30% $\pm$ 0.01%	+3.01% $\pm$ 0.02%	+6.11% $\pm$ 0.02%	+3.04% $\pm$ 0.01%
MDP	+7.11% $\pm$ 0.05%	+2.71% $\pm$ 0.03%	+7.89% $\pm$ 0.05%	+3.13% $\pm$ 0.04%	+7.53% $\pm$ 0.03%	+3.19% $\pm$ 0.03%
DDQN	+6.68% $\pm$ 0.04%	+3.19% $\pm$ 0.04%	+7.75% $\pm$ 0.06%	+4.06% $\pm$ 0.05%	+7.62% $\pm$ 0.04%	+4.58% $\pm$ 0.05%
MFOD	+6.62% $\pm$ 0.03%	+3.71% $\pm$ 0.02%	+7.91% $\pm$ 0.04%	+4.01% $\pm$ 0.02%	+7.32% $\pm$ 0.02%	+4.60% $\pm$ 0.01%
CoRide-	+9.27% $\pm$ 0.04%	+4.23% $\pm$ 0.03%	+8.73% $\pm$ 0.03%	+4.35% $\pm$ 0.02%	+9.06% $\pm$ 0.03%	+4.23% $\pm$ 0.04%
CoRide	+9.80% $\pm$ 0.04%	+4.81% $\pm$ 0.05%	+8.94% $\pm$ 0.06%	+4.89% $\pm$ 0.04%	+9.23% $\pm$ 0.05%	+5.19% $\pm$ 0.04%

Table 2. Table 2. Performance comparison of competing methods in terms of AST and TNF with three different discounted rate s (DR). The numbers in Trajectory denote gridID in Figure  7 and its color denotes the district it located in. O and W mean the vehicle is On-service and Waiting at the current grid. Also, we use underlined number to present fleet management.

DR	20%			30%			40%
Metrics	Trajectory	AST	TNF	Trajectory	AST	TNF	Trajectory	AST	TNF
RES	13, 9, W, 14, W, 13, 8, 2, 7, 11	8	8	13, W, 14, W, W, 13, 8, W, 7, 11	6	6	13, W, 14, W, W, W, W, 19, O, 9	5	4
REV	O, O, 15, W, 20, O, O, O, O, 11	9	3	O, O, 15, W, W, W, O, O, O, 11	7	2	O, O, 15, W, W, W, 20, W, O, 14	6	3
CoRide	13, 9, W, O, 0, 4, O, 2, O, 5	9	6	13, W, O, 11, W, W, O, O, 0, 5	7	4	13, W, W, O, 17, W, W, O, 0, 3	6	3
CoRide+	13, 9, W, O, 0, 4, O, 2, O, 5	9	6	8, 3, 0, 2, O, 4, O, 2, O, 5	8	5	8, 3, 0, 2, O, 4, O, 2, O, 5	8	5

Equations

$$E=-k_B\cdot \sum _i{\rho }_i\log {\rho }_i:=-k_B\cdot {\rho }_0\log {\rho }_0$$

$${\rho }_0=\frac{\text{min}(N_v,N_o)\times \text{min}(N_v,N_o)}{N_v\times N_o}=\frac{N_v\times N_v}{N_v\times N_o}=\frac{N_v}{N_o}$$

$$h_t^M,{\hat{g}}_t=\text{RNN}(s_t^M,h_{t-1}^M;{\theta }^{rnn});g_t={\hat{g}}_t/||{\hat{g}}_t||$$

$$r_t^M=r_{ADI}+r_{ORR}$$

$$r_{ORR}=\sum _{grid}(E-\bar{E}{)}^2+\sum _{area}{D}_{KL}({\mathcal{P}}_t^o\Vert {\mathcal{P}}_t^v)$$

$$r_t^W=\frac{1}{c}\sum _{i=1}^c{d}_{cos}(o_t^W-o_{t-i}^W,g_{t-i}),$$

$$f_{{\theta }^W}:o_t^W\to {\omega }_t,h_t^W$$

$$scor{e}_i={\omega }_t^T{e}_i$$

$$\text{Selected-}k=\frac{\text{exp}(scor{e}_i/\tau )}{{\sum }_{i=1}^M\text{exp}(scor{e}_i/\tau )}$$

$$h_{t-1}^{ij}=(h_{t-1}^i{W}_T)\cdot (h_{t-1}^j{W}_S{)}^T$$

$${\alpha }_{t-1}^{ij}=\text{softmax}(h_{t-1}^{ij})=\frac{\text{exp}(h_{t-1}^{ij}/\iota )}{{\sum }_{j\in {\mathcal{N}}_i}\text{exp}(h_{t-1}^{ij}/\iota )}$$

$$m_{t}^{i}=\sigma\Big{(}W_{q}\cdot\big{(}\cfrac{1}{H}\sum_{n=1}^{n=H}\sum_{j\in\mathcal{N}_{i}}\alpha_{t-1}^{ijn}(h_{t-1}^{j}W^{n}_{C})\big{)}+b_{q}\Big{)}$$

$$Q^{\ast }(o_t,a_t)={\mathbb{E}}_{o_{t+1}}[r_t+\gamma \underset{a_{t+1}}{\max }{Q}^{\ast }(o_{t+1},a_{t+1})|o_t,a_t]$$

$$L({\theta }^{\mu })={\mathbb{E}}_{s_t,a_t,r_t,o_{t+1}}[(y_t-Q(o_t,a_t;{\theta }^{\mu }){)}^2]$$

Full text

Hierarchical Reinforcement Learning; Multi-agent Reinforcement Learning; Ride-Hailing; Order Dispatching; Fleet Management

††submissionid: lp0949††journalyear: 2019††copyright: acmcopyright††conference: The 28th ACM International Conference on Information and Knowledge Management; November 3–7, 2019; Beijing, China††booktitle: The 28th ACM International Conference on Information and Knowledge Management (CIKM’19), November 3–7, 2019, Beijing, China††price: 15.00††doi: 10.1145/3357384.3357978††isbn: 978-1-4503-6976-3/19/11††ccs: Computing methodologies Multi-agent reinforcement learning††ccs: Theory of computation Multi-agent reinforcement learning††ccs: Applied computing Transportation

**ACM Reference Format:

**Jiarui Jin, Ming Zhou, Weinan Zhang, Minne Li, Zilong Guo, Zhiwei Qin, Yan Jiao, Xiaocheng Tang, Chenxi Wang, Jun Wang, Guobin Wu and Jieping Ye. 2019. CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM ’19), November 3–7, 2019, Beijing, China. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3357384.3357978

1. Introduction

Online ride-hailing platforms such as Uber and Didi Chuxing have substantially transformed our lives by sharing and reallocating transportation resources to highly promote transportation efficiency. In a general view, there are two major decision-making tasks for such ride-hailing platforms, namely (i) order dispatching, i.e., to match the orders and vehicles in real time to directly deliver the service to the users (Zou et al., 2013; Zhang et al., 2017; Seow et al., 2010), and (ii) fleet management, i.e., to reposition the vehicles to certain areas in advance to prepare for the later order dispatching (Lin et al., 2018; Oda and Tachibana, 2018; Simao et al., 2009).

Apparently, the decision-making of matching an order-vehicle pair or repositioning a vehicle to an area needs accounting for the future situation of the vehicle’s location and the orders nearby. Thus, much of work has modeled order dispatching and fleet management as a sequential decision-making problem and solved it with reinforcement learning (RL) (Wang et al., 2018b; Xu et al., 2018; Lin et al., 2018; Tang and Qin, 2018). Most of the previous work deals with either order dispatching or fleet management without regarding the high correlation of these two tasks, especially for large-scale ride-hailing platforms in large cities, which leads to sub-optimal performance. In order to achieve near-optimal performance, inspired by thermodynamics, we simulate the whole ride-hailing platform as dispatch (order dispatching) and reposition (fleet management). As illustrated in Figure 1, we resemble vehicle and order as different molecules and aim at building up the system stability via reducing their number by dispatch and reposition. To address this complex criterion, we provide two novel views: (i) interconnecting order dispatching and fleet management, and (ii) joint considering intra-district (grid-level) and inter-district (district-level) allocation. With such a practical motivation, we focus on modeling joint order dispatching and fleet management with multi-scale decision-making system. There are several significant technical challenges to learn highly efficient allocation policy for the real-time ride-hailing platform:

Large-scale Agents. A fundamental question in any ride-hailing system is how to deal with a large number of orders and vehicles. One alternative is to model each available vehicle as an agent (Oda and Tachibana, 2018; Xu et al., 2018; Wei et al., 2018). However, such setting needs to maintain thousands of agents interacting with the environment, which brings a huge computational cost. Instead, we utilize the region grid world (as will be further discussed in Figure 2), which regards each region as an agent, and naturally model ride-hailing system in a hierarchical learning setting. This formulation allows decentralized learning and control with distributed implementation.

Immediate & Future Rewards. A key challenge in seeking an optimal control policy is to find a trade-off between immediate and future rewards in terms of accumulated driver income (ADI). Greedily matching vehicles with long-distance orders can receive high immediate gain at a single order dispatching stage, but it would harm order response rate (ORR) and future revenue especially during rush hour because of its long drive time and unpopular destination. Recent attempts (Xu et al., 2018; Wei et al., 2018; Oda and Tachibana, 2018) deployed RL to combine instant order reward from online planning with future state-value as the final matching value. However, the coordination between different regions is still far from optimal. Inspired by hierarchical RL (Vezhnevets et al., 2017), we introduce the geographical hierarchical structure of region agents. We treat large district as manager agent and small grid as worker agent, respectively. The manager operates at a lower spatial and temporal dimension and sets abstract goals which are conveyed to its workers. The worker takes specific actions and interacts with environment coordinated with manager-level goal and worker-level message. This decoupled structure facilitates very long timescale credit assignment (Vezhnevets et al., 2017) and guarantees balance between immediate and future revenue.

Heterogeneous & Variant Action Space. Traditional RL models require a fixed action space (Mnih et al., 2013). If we model picking an order as an RL action, there is no guarantee of a fixed action space as the available orders keep changing. Zhang et al. (2017) proposed to learn a state-action value function to evaluate each valid order-vehicle match, then use a combinatorial optimization method such as Kuhn-Munkres (KM) algorithm (Munkres, 1957) to filter the matches. However, such a method faces another important challenge that order dispatching and fleet management are different tasks, which results in heterogeneous action spaces. To address this issue, we redefine action as the weight vector for ranking orders and fleet management, where the fleet controls are regarded as fake orders, and all the orders are ranked and matched with vehicles in each agent. Thus, it bypasses the issue of heterogeneous and variant action space as well as high computational costs.

Multi-Scale Ride-Hailing. Xu et al. (2018) introduced a policy evaluation based RL method to learn the dynamics for each grid. As its result shows, orders and vehicles often centralize at different districts (e.g. uptown and downtown in Figure 1). How to combine large hotspots in the city (inter-district) with small ones in districts (intra-district) is another challenge without much attention. In order to take both inter-district and intra-district allocation into consideration, we adopt and extend attention mechanism in a hierarchical way (as will be further discussed in Figure 3). Compared with learning value function for each grid homogeneously (Xu et al., 2018), this attention-based structure can not only capture the impacts of neighbor agents, but also learn to distinguish the key grid and district in worker (grid) and manager (district) scales respectively.

Wrapping all modules together, we propose CoRide, a hierarchical multi-agent reinforcement learning framework to resolve the aforementioned challenges. The main contributions are listed as follows:

• •

We propose a novel framework that learns to collaborate in hierarchical multi-agent setting for ride-hailing platform.

• •

We conduct extensive experiments based on real-world data of multiple cities, as well as analytic synthetic data, which demonstrate that CoRide provides superior performance in terms of ADI and ORR in the task of city-wide hybrid order dispatching and fleet management over strong baselines.

• •

To the best of our knowledge, CoRide is the first work (i) to apply the hierarchical reinforcement learning on ride-hailing platform; (ii) to address the task of joint order dispatching and fleet management of online ride-hailing platforms; (iii) to introduce and study multi-scale ride-hailing task.

This structure conveys several benefits: (i) In addition to balancing long-term and short-term reward, it also facilitates adaptation in a dynamic real-world situation by assigning different goals to worker. (ii) Instead of considering all of the matches between available orders and vehicles globally, these tasks are distributed to each worker and manager agent and fulfilled in a parallel way.

2. Related Work

Decision-making for Ride-hailing. Order dispatching and fleet management are two major decision-making tasks for online ride-hailing platforms, which have acquired much attention from academia and industry during the recent few years.

To tackle these challenging transportation problems, automatically ruled-based approaches addressed order dispatching problem with either centralized or decentralized settings. To improve global performance, Zhang et al. (2017) proposed a novel model based on centralized combinatorial optimization by concurrently matching multiple vehicle-order pairs within a short time window. However, this approach needs to compute all available vehicle-order matches and requires feature engineering, which would be infeasible and prevent it to be adopted in the large-scale taxi-order dispatching situation. With the decentralized setting, Seow et al. (2010) addressed this problem with a collaborative multi-agent taxi dispatching system. However, this method requires rounds of direct communications between agents, so it is limited to a local area with a small number of vehicles.

Instead of rule-based approaches, which require additional handcrafted heuristics, the current trending direction is to incorporate reinforcement learning algorithms in complicated traffic management problems. Xu et al. (2018) proposed a learning and planning method based on reinforcement learning to optimize vehicle utilization and user experience in a global and more farsighted view. In (Oda and Tachibana, 2018), the authors leveraged the graph structure of the road network and expanded distributed DQN formulation to maximize entropy in the agents’ learning policy with soft Q-learning, to improve performance of fleet management. Wei et al. (2018) introduced a reinforcement learning method, which takes the uncertainty of future requests into account and can make a look-ahead decision to help the operator improve the global level-of-service of a shared-vehicle system through fleet management. To capture the complicated stochastic demand-supply variations in high-dimensional space, Lin et al. (2018) proposed a contextual multi-agent actor-critic framework to achieve explicit coordination among a large number of agents adaptive to different contexts in fleet management system.

Different from all aforementioned methods, our approach is the first, to the best of our knowledge, to consider the joint modeling of order dispatching and fleet management and also the only current work introducing and studying the multi-scale ride-hailing task.

Hierarchical Reinforcement Learning. Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve tasks with long-term dependency or multi-level interaction patterns (Dayan and Hinton, 1993; Dietterich, 2000). Recent works have suggested that several interesting and standout results can be induced by training multi-level hierarchical policy in a multi-task setup (Frans et al., 2017; Sigaud and Stulp, 2018) or implementing hierarchical setting in sparse reward problems (Riedmiller et al., 2018; Vezhnevets et al., 2017).

The options framework (Stolle and Precup, 2002; Precup, 2000; Sutton et al., 1999) formulates the problem with a two-level hierarchy, where the low-level - option - is a sub-policy with a termination condition. Since traditional options framework suffers from prior knowledge on designing options, jointly learning high-level policy with low-level policy has been proposed by (Bacon et al., 2017). However, this actor-critic HRL approach needs to either learning sub-policies for each time step or one sub-policy for the whole episode. Therefore, the performance of the whole module often prone to learning useful sub-policies. To guarantee gaining effective sub-policies, one alternative direction is to provide auxiliary rewards for low-level policy: hand-designed rewards based on prior domain knowledge (Kulkarni et al., 2016; Tessler et al., 2017) or mutual information (Florensa et al., 2017; Daniel et al., 2012; Kong et al., 2017). Given having access to one well-designed and suitable reward is often a luxury, Vezhnevets et al. (2017) proposed FeUdal Networks (FuN), where a generic reward is utilized for low-level policy learning, thus avoid designing hand-craft rewards. Several works extend and improve FuN with off-policy training (Nachum et al., 2018a), form of hindsight (Levy et al., 2018) and representation learning (Nachum et al., 2018b).

Our work is also developed from FuN (Vezhnevets et al., 2017), originally inspired by feudal RL (Dayan and Hinton, 1993). FuN employs only one pair of manager and worker and connects them with a parameterized goal and intrinsic reward. Instead, we model CoRide with multiple managers. Unlike our method, in FuN the manager and worker modules are set to one-to-one, share the same observation, and operate at the different temporal but same spatial resolution. In CoRide, there are multiple workers learning to collaborate under one manager while the managers are also coordinating. The manager takes a joint observation of all workers, and each worker produces action based on specific observation and sharing goal. In other words, FuN (Vezhnevets et al., 2017) is actually a special case of CoRide, where a single manager along with its only worker is employed. Stepping on this one-to-many setting, the manager can not only operate with long timescale credit but act at a lower spatial resolution. Recently, Ahilan and Dayan (2019) introduced a novel architecture named FMH for cooperation in multi-agent RL. Different from this proposed method, CoRide not only extends the scale of the multi-agent environment but also facilitates communication through multi-head attention mechanism, which computes influences of interactions and differentiates the impacts to each agent. Yet, the majority of current HRL methods require careful task-specific design, making them difficult to apply in real-world scenarios (Nachum et al., 2018a). To the best of our knowledge, CoRide is the first work to apply hierarchical reinforcement learning on the ride-hailing problem.

3. Problem Formulation

We formulate the problem of controlling large-scale homogeneous vehicles in online ride-hailing platforms, which combines order dispatching system with fleet management system with the goal of maximizing the city-level ADI and ORR. In practice, vehicles are divided into two groups: order dispatching (OD) group and fleet management (FM) group. For OD group, we match these vehicles with available orders pair-wisely; whereas for FM group, we need to reposition them to the locations or dispatch orders to them (same as OD group). The illustration of the problem is shown in Figure 2. We use the hexagonal-grid world to represent the map and take a grid as an agent. Considering that only orders within the pick-up distance can be dispatched to vehicle, we set distance between grids based on the pick-up distance. Given that, in our setting, vehicles in the same spatial-temporal node are homogeneous, i.e., the vehicles located at the same grid share the same setting. As such, we can model order dispatching as a large-scale parallel ranking task, where we rank orders and match them with homogeneous vehicles in each grid. The fleet control for fleet management, i.e. repositioning the vehicle to neighbor grids or staying at the current grid, is treated as fake orders (as will be further discussed in Section 6) and conducted in the ranking procedure as same as order dispatching.

Since each agent can only reposition vehicles located in the managing grid, we propose to formulate the problem using Partially Observable Markov Decision Process (POMDP) (Spaan, 2012) in a hierarchical multi-agent reinforcement learning setting for both order dispatching and fleet management. Thus, we decompose the original complicated tasks into many local ones and transform a high-dimensional problem into multiple low-dimensional problems.

Formally, we model this task as a Markov game $\mathcal{G}$ for $N$ agents, which is defined by a tuple $\mathcal{G}=(N,\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, where $N,\mathcal{S}$, $\mathcal{A}$, $\mathcal{P}$, $\mathcal{R}$, $\gamma$ are the number of agents, set of states, set of actions, state transition probability, reward function, and a future reward discount factor, respectively. The definitions of major components are as follows.

Agent. We consider each region cell as an agent identified by $i\in\mathcal{I}$, where $\mathcal{I}=\{i\mid i=1,\dots,N\}$. In detail, a single grid represents a worker agent, a district containing multiple grids represents a manager agent. An example is presented in Figure 2. Each individual grid serves as a worker agent with 6 neighbor grids, as shown in the same color, composes a manager agent. Note that although the number of vehicles and orders varies over time, the number of agents is fixed.

State $s_{t}\in\mathcal{S}$, Observation $o_{t}\in\mathcal{O}$. Although there are two different types of agents - manager and worker, their observations only differ in scale. Observation of each manager is actually a joint observation of its workers. At each timestep $t$, agent $i$ draws private observations $o^{i}_{t}\in\mathcal{O}$ correlated with the environment state $s_{t}\in\mathcal{S}$. In our setting, the state input used in our method is expressed as $\mathcal{S}=\langle N_{v},N_{o},E,N_{f},D_{o}\rangle$, where inner elements represent the number of vehicles, number of order, entropy, number of vehicles in FM group and distribution of order features (e.g. price, duration) in current grid respectively. Note that both dispatching and repositioning belong to resource allocation similar to the thermodynamic system (Figure 1), and once order dispatching or fleet management occurs, dispatched or fleeted items slip out of the system. Namely, only idle vehicles and available orders can contribute to disorder and unevenness of the ride-hailing system. Therefore, we introduce and extend the concept of entropy here, defined as:

[TABLE]

where $k_{B}$ is a Boltzmann constant, and $\rho_{i}$ means probability for each state: $\rho_{1}$ for dispatched and fleeted, $\rho_{0}$ elsewhere. As aforementioned analysis, once order and vehicle combined as order-vehicle pairs, both order and vehicle are out of the ride-hailing platform. Therefore, we choose to ignore items at state 1 ($\rho_{1}$) and compute $\rho_{0}$ as proportion of available order-vehicle pairs in all potential pairs:

[TABLE]

which is conditioned in $N_{v}<N_{o}$ situation and straightforward to transform to other situations.

Action $a_{t}\in\mathcal{A}$, State Transition $\mathcal{P}$. In our hierarchical RL setting, manager’s action is to generate abstract and intrinsic goal to its workers, and each worker needs to provide a ranking list of relevant real orders (OD) and fleet control served as fake orders (FM). Thus, the action of worker is defined as the weight vector for the ranking features. Changing an action of the worker means to change the weight vector for the ranking features (as will be further discussed in Section 6). Each timestep the whole multi-agent system produces a joint action for each manager and worker $a_{t}\in\mathcal{A}$, where $\mathcal{A}:=\mathcal{A}_{1}\times...\times\mathcal{A}_{N}$, which induces a transition in the environment according to the state transition $\mathcal{P}(s_{t+1}|s_{t},a_{t})$.

Reward $\mathcal{R}$. Like previous hierarchical RL settings (Vezhnevets et al., 2017), only manager interacts with the environment and receives feedback from it. This extrinsic reward function determines the direction of optimization and is proportional to both immediate profit and potential value; while the intrinsic reward is set to encourage the worker to follow the instruction from the manager. The details will be further discussed in Eq. (4) and Eq. (6).

More specifically, we give a simple example based on the above problem setting in Figure 2. At time $t=0$, the worker agent 0 ranks available real orders and potential fake orders for fleet control, and selects the Selected-2 (as will be further discussed in Eq. (9)) options: a real order from grid 0 to grid 17, a fake order from grid 0 to grid 5. After the driver finished, the manager agent, whose sub-workers maintain the worker agent 0, received corresponding reward.

4. Methodologies

4.1. Overall Architecture

As shown in Figure 3, CoRide employs two layers of agents, namely the layer of manager agents and the layer of worker agents. Each agent is associated with a communication component for exchanging messages. As both agent and decision-making process conduct in a hierarchical way, multi-head attention mechanism served for communication is extended into multi-layer setting.

Compared with traditional one-to-one manager-worker control in hierarchical RL (Vezhnevets et al., 2017), we design one-to-many manager-worker pattern, extend the scale, and learn to collaborate on two layers of agents. The manager internally computes a latent state representation $h_{t}^{M}$ as an input to the manager-level attention network, and outputs a goal vector $g_{t}$. The worker produces action and input for worker-level attention conditioned on its private observation $o_{t}^{W}$, peer message $m_{t-1}^{W}$, and the manager’s goal $g_{t}$. The manager-level and worker-level attention networks share the same architecture introduced in Section 4.4. The details and training procedure for manager and worker are given in following parts.

4.2. Manager Module

The architecture of the manager module is presented in Figure 4. The manager network is a two layer Preceptron (MLP) and a dilated RNN (Vezhnevets et al., 2017). Note that the structure of CoRide and formula of the RNN enable manager operate both at lower spatial resolution via taking joint observation of its workers and lower temporal resolution via dilated convolutional network (Yu and Koltun, 2015).

At timestep $t$, the agent receives an observation $o^{M}_{t}$ from the environment and feeds into the dilated RNN with peer messages $m^{M}_{t-1}$. Goal $g_{t}$ and input for manager-level attention $h^{M}_{t}$ are generated as output of the RNN, governed by the following equations:

[TABLE]

where $\theta^{rnn}$ is the parameters of the RNN network. The environment responds with a new observation $o^{M}_{t+1}$ and a scalar reward $r_{t}$. The goal of the agent is to maximize the discounted return $\mathcal{R}_{t}=\sum_{k=0}^{\infty}\gamma^{k}r^{M}_{r+k+1}$ with $\gamma\in[0,1]$. Specifically, in the ride-hailing setting, we design our global reward taking both ADI and ORR into account, which can be formulated as:

[TABLE]

where $r_{ADI}$ denotes accumulated driver income, computed according to the price of each served order; while $r_{ORR}$ encourages ORR, and is calculated with several correlative factors as:

[TABLE]

where $E,\bar{E}$ are the manager’s entropy and global average entropy respectively. Area, different from grid, often means a certain region which needs to be taken more care of. In our experiment, we select several grids whose entropy largely differs from the average as the area. $D_{KL}(\mathcal{P}^{o}_{t}\|\mathcal{P}^{v}_{t})$ denotes Kullback-Leibler (KL) divergence which shows the margin between vehicle and order distributions of certain area at timestep $t$. $\mathcal{P}^{o}_{t}$ and $\mathcal{P}^{v}_{t}$ are realized with Poisson distribution, a common distribution for vehicle routing (Ghiani et al., 2003) and arriving (Lord et al., 2005). In practice, this distribution parameters can be estimated from real trip data by the mean and std of orders and vehicles in each grid at each timestep. Such a combined ORR reward design helps optimization both globally and locally.

4.3. Worker Module

We adopt the goal embedding from Feudal Networks (FuN) (Vezhnevets et al., 2017) in our worker framework (see Figure 5), where $w_{t}$ is generated as goal-embedding vector via linear projection. At each timestep $t$, the agent receives an observation $o_{t}^{W}$ from the environment and feeds into a regular RNN with peer message $m^{W}_{t-1}$. As Figure 5 shows, the output of RNN $u_{t}^{W}$ together with $w_{t}$ generates the primitive action - ranking weight vector $\omega_{t}$.

Given that worker needs to be encouraged to follow the goal generated by its manager, we adopt the intrinsic reward proposed by (Vezhnevets et al., 2017), defined as:

[TABLE]

where $d_{cos}(\alpha,\beta)=\alpha^{T}\beta/(|\alpha|\cdot|\beta|)$ is the cosine similarity between two vectors. Notice that $g_{t}$ now represents an advantageous direction in the latent state space at a horizon $c$ (Vezhnevets et al., 2017). Such intrinsic reward design would provide directional shift for $worker$s to follow.

Different from traditional FuN (Vezhnevets et al., 2017), procedure of our worker module produces action consists of two steps: (i) parameter generating, and (ii) action generating, inspired by (Zhao et al., 2017). We utilize state-specific scoring function $f_{\theta^{W}}$ in parameter generating setup to map the current state $o^{W}_{t}$ to a list of weight vectors $\omega_{t}$ as:

[TABLE]

which is calculated based on nerual network shown in Figure 5. In action generating setup, note that it is straightforward to extend linear relations with non-linear ones, we formulate that the scoring function parameter $\omega_{t}$ and the ranking feature $e_{i}$ for order $i$ as:

[TABLE]

The detailed formulation of $e_{i}$ will be discussed in Section 5. Then, we build and add real orders and potential fleet control - repositioning to neighbor grids and staying at the current grid - as fake orders into item space $\mathcal{I}$. After computing scores for all available options in $\mathcal{I}$, instead of directly ranking and selecting Top-$k$ items for order dispatching and fleet management, we adopt Boltzmann softmax selector to generate Selected-$k$ items:

[TABLE]

where $k=\text{min}(N_{v},N_{o})$, $\tau$ denotes temperature hyper-parameter to control the exploration rate, and $M$ is the number of scored order candidates. In practice, we set the initial temperature as 1.0, then gradually reduce the temperature until 0.01 to limit exploration. This approach not only equips the action selection procedure with controllable exploration but also diversify the policy’s decision to avoid choosing groups of vehicles fleeted to the same destination.

4.4. Multi-head Attention for Coordination

Note that manager and worker share the same setting of multi-head attention mechanism, agent in this subsection can represent either of them. We utilize $h^{i}_{t-1}$ to denote the cooperating information for $i$-th agent generated from RNN at $t$-1, and extend self-attention mechanism to learn to evaluate each available interaction as:

[TABLE]

where $h^{i}_{t-1}W_{T},h^{j}_{t-1}W_{S}$ are embedding of messages from target agent and source agent respectively. We can model $h_{t-1}^{ij}$ as the value of communication between $i$-th agent and $j$-th agent. To retrieve a general attention value between source and target agents, we further normalize this value in neighborhood scope as:

[TABLE]

where $\mathcal{N}_{i}$ is the neighborhood scope: the set of communication available for target agent, and $\iota$ denotes temperature factor. To jointly attend to the neighborhood from different representation subspaces at different grids, we leverage multi-head attention as in previous work (Vaswani et al., 2017; Veličković et al., 2017; Wei et al., 2019; Zhang et al., 2019) to extend the observation as:

[TABLE]

where $H$ is the number of attention heads, and $W_{T},W_{S},W_{C}$ are multiple sets of trainable parameters. Thus, peer message $m_{t}$ is generated and will be feed into the corresponding module to produce the cooperative information $h_{t}$. We present the overall CoRide for joint order dispatching and fleet management in Algorithm 1.

4.5. Training

As described in Algorithm 1, managers generate specific goals based on their observations and peer messages (line 2). The workers under the manager generate the weight vector according to private observation and sharing goal (line 4). We then build a general item space $\mathcal{I}$ for order dispatching and fleet management (line 5), and rank items in $\mathcal{I}$ (line 6). Considering that our action is conditional to the minimum of the number of vehicles and orders, we generate Selected-$k$ items as the final action (line 7).

We extend learning approach from FuN (Vezhnevets et al., 2017) and HIRO (Nachum et al., 2018a) to train manager and worker module in the similar way. In CoRide, we utilize DDPG algorithm (Lillicrap et al., 2015) to train the parameters for both manager and worker module for following reasons. Classically, the critic is designed to leverage an approximator, to learn an action-value function $Q(o_{t},a_{t})$, and to direct the actor updating its parameters. The optimal action-value function $Q^{*}(o_{t},a_{t})$ should follow the Bellman equation (Bellman, 2013) as:

[TABLE]

which requires $|\mathcal{A}|$ evaluations to select the optimal action. This prevents Eq. (13) to be adopted in real-world scenario, e.g. ride-hailing setting, with enormous state and action spaces. However, the actor architecture proposed in Section 4.3 generates a deterministic action for critic. Furthermore, Lillicrap et al. (2015) proposed a flexible and practical method to use an approximator function to estimate the action-value function, i.e. $Q(o,a)\thickapprox Q(o,a;\theta^{\mu})$. In practice, we refer to leverage DQN: a neural network function approximator can be trained by minimizing a sequence of loss functions $L(\theta^{\mu})$ as:

[TABLE]

where $y_{t}=\mathbb{E}_{o_{t+1}}[r_{t}+\gamma Q^{\prime}(o_{t+1},a_{t+1};\theta^{\mu^{\prime}})|o_{t},a_{t}]$ is the target for the current iteration. According to the aforementioned analysis, the general training algorithm for the manager and worker module is presented in Algorithm 2.

5. Simulator

The trial-and-error nature of reinforcement learning requires a dynamic simulation environment for training and evaluation. Thus, we adopt and extend the grid-based simulator designed by Lin et al. (2018) to joint order dispatching and fleet management.

5.1. Data Description

The real world data provided by Didi Chuxing††††Similar dataset supported by Didi Chuxing can be found via GAIA open dataset (https://outreach.didichuxing.com/research/opendata/en/). includes order information and trajectories of vehicles in the central area of three large cities with millions of orders in four consecutive weeks. Data of each day contains million-level orders and tens of thousands vehicles in each city. The order information includes order price, origin, destination, and duration. The trajectories contain the positions (latitude and longitude) and status (on-line, off-line, on-service) of all vehicles every few seconds. As the radius of grid is approximate 1.3 kilometers, the central area of the city is covered by a hexagonal grids world consisting of 182, 126, 112 grids in three cities respectively. In order to adapt to the grid-based simulator, we utilize unique gridID to represent position information.

5.2. Simulator Design

In the grid-based simulator, the city is covered by a hexagonal grid-world as illustrated in Figure 2. At each timestep $t$, the simulator provides an observation $o_{t}$ with a set of idle vehicles and a set of available orders including real orders and aforementioned fake orders for fleet control. All such fake orders share the same attributes as real orders, except that some of attributes are set stationary like price. All these real orders are generated by bootstrapping from real-world dataset introduced above. More specifically, suppose the current timestep of simulator is $t$, we randomly sample real orders occuring in the same period, i.e. happening between $t_{\Delta}\times t$ to $t_{\Delta}\times(t+1)$, where $t_{\Delta}$ denotes timestep interval. In practice, we set sampling rate 100%. Like orders, vehicles are set online and offline alternatively according to a distribution learned from real-world dataset via maximum likelihood estimation. Each order feature, i.e. ranking feature $e_{i}$ in Eq. (8), includes the origin gridID, the destination gridID, price, duration and the type of order indicating real or fake order; while each vehicle takes its gridID as a feature, and vehicles located at the same grid are regarded as homogeneous. Moreover, as the travel distance between neighboring grids is approximately 1.3 kilometers and timestep interval $t_{\Delta}$ is 10 minutes, we assume that vehicles will not automatically move to other grids before taking another order. The ride-hailing platform then provides an optimal list of vehicle-order pairs according to current policy. After receiving the list, the simulator will return a new observation $o_{t+1}$ and a list of order fees. Stepping on this feedback, rewards $r_{t}$ for each agent will be calculated and the corresponding record $(o_{t},a_{t},r_{t},o_{t+1})$ will be stored into a replay buffer. The whole network parameters will be updated using a batch of samples from replay buffer.

The effectiveness of the grid-based simulator is evaluated based on the calibration against the real data in term of the most important performance measurement: accumulated driver income (ADI) (Lin et al., 2018). The coefficient of determination $r^{2}$ between simulated ADI and real ADI is 0.9331 and the Pearson correlation is 0.9853 with $p$-value $p<0.00001$.

6. Experiment

In this section, we conduct extensive experiments to evaluate the effectiveness of our proposed method in joint order dispatching and fleet management environment. Given that there are no published methods fitting our task. Thus, we first compare our proposed method with other models either widely used in the industry or published as academic papers based on a single order dispatching environment. Then, we further evaluate our proposed method in joint setting and compare with its performance in single setting.

6.1. Compared Methods

As discussed in (Lin et al., 2018), learning-based methods, currently regarded as state-of-the-art methods, usually outperform rule-based methods. Thus, we employ 6 learning-based methods and random method as the benchmark for comparison in our experiments.

• •

RAN: A random dispatching algorithm considering no additional information. It only assigns idle vehicles with available orders randomly at each timestep.

• •

DQN: Li et al. (2019) conducted action-value function approximation based on $Q$-network. The $Q$-network is parameterized by a MLP with four hidden layers and we adopt the ReLU activation between hidden layers and to transform the final linear output of $Q$-network.

• •

MDP: Xu et al. (2018) implemented dispatching through a learning and planning approach: each vehicle-order pair is valued in consideration of both immediate rewards and future gains in the learning step, and dispatch is solved using a combinatorial optimizing algorithm in planning step.

• •

DDQN: Wang et al. (2018b) introduced a double-DQN with spatial-temporal action search. The network architecture is similar to the one described in DQN except that a selected action space is utilized and network parameters are updated via double-DQN.

• •

MFOD: Li et al. (2019) modeled the order dispatching problem with MFRL (Yang et al., 2018) and simplified the local interactions by taking an average action among neighborhoods.

• •

CoRide: Our proposed model as detailed in Section 4.

• •

CoRide-: In order to further evaluate performance for hierarchical setting and agent communication, we set CoRide without multi-head attention mechanism as one of the baselines.

6.2. Result Analysis

For all learning methods, following (Li et al., 2019), we run 20 episodes for training, store the trained model periodically, and conduct the evaluation on the stored model with 5 random seeds. We compare the performance of different models regarding two criteria, including ADI, computed as the total income in a day, and ORR, calculated by the number of orders taken divided by the number of orders generated.

Experimental Results and Analysis. As shown in Table 1, the performance surpasses the state-of-the-art models like DDQN and industry deployed model like MDP. DDQN along with DQN mainly limits in lack of interaction and cooperation in the multi-agent environment. MDP mainly focuses on order price but ignores other features of order like duration, which makes against finding a balance between getting higher income per order and taking more orders. Instead, our proposed method achieves higher growths in term of ADI not only by considering every feature of each order concurrently but through learning to collaborate hierarchically. MFOD trys to capture dynamic demand-supply variations by propagating many local interactions between vehicles and the environment among mean field. Note that the number and information of available grid are relatively stationary while the number and feature of active vehicles are more dynamic. Thus, CoRide, which takes grid as agent, is more likely and easier to learn to cooperate from interaction between agents.

Apart from cooperation, multi-head attention network also enables CoRide to capture demand-supply dynamics from both district (manager) and grid (worker) scale (as will be further discussed in Figure 6). Such a novel combined scale setting facilitates CoRide both effectively and efficiently.

Visualization Analysis. Except for quantitive results, we also analyze whether the learned multi-head attention network can capture the demand-supply relation (see Figure 6(b)) through visualization. As shown in Figure 3, our communication mechanism conducts in a hierarchical way: attention among the managers communicates and learns to collaborate abstractly and globally while peers in worker-layer operate and determine key grid locally.

The values of several example managers and a group of workers belonging to the same manager are visualized in Figure 6(a). By taking a closer look at Figure 6, we can observe that the area with high demand-supply indeed centralized at certain places, which has been well captured in manager-scale. Such district-level attention value allows precious vehicles to be dispatched efficiently in a global view. Apart from manager-scale one, multi-head attention network also provides worker-scale attention value, which focuses on local allocation. Stepping on this multi-scale dispatching system design, CoRide could operate as a microscope, where coarse and fine focuses work together to obtain precise action.

Ablation Study. In this subsection, we evaluate the effectiveness of components of CoRide. Notice that manager and worker modules serve as key components and are integrated through the hierarchical multi-agent architecture, as Figure 3 shows. Thus, we choose to investigate the performance of multi-head attention network here and set CoRide- as a variation of proposed method. As shown in the last two rows in Table 1, CoRide- achieves significant advantages over the aforementioned baselines, especially in City A. Similar results occur with CoRide. This phenomenon can be explained from the fact that City A is the largest one according to Section 5.1, which requires frequent and large numbers of transportations among regions. Multi-scale guidance via multi-head attention network and hierarchical multi-agent architecture is therefore potentially helpful, especially at a large-scale case.

Case Study.

The above experimental results show that the success rate of our model is significantly better than others in single dispatching task. In order to evaluate the performance of CoRide in joint dispatching and repositioning. Also, to further differ the formulations of the models, we constructed a synthetic dataset containing 3 districts with 21 grids, as showed in Figure 7. All these synthetic datasets are obtained via sampling real-world dataset supported by Didi Chuxing. More concretely, order distributions of all grids are sampled from the average distribution in real world dataset. Namely, order distributions in each grid are homogeneous. In order to differ downtown areas from uptown areas, we introduce sampling rate here. The sampling rate for each grid denotes popular rate in the real world. We set downtown (red grids in Figure 7) with stationary sampling rate 100%. The other regions are sampled with $\emph{sampling rate}:=100\%-\emph{discounted rate}$ for yellow district and $100\%-2\times\emph{discounted rate}$ for green district. Specifically we set discounted rate as $20\%$, $30\%$ and $40\%$ respectively, and further verify our proposed model by comparing against following 3 methods:

• •

RES: This response-based method aims to achieve higher ORR, which corresponds to Total Number of Finished orders (TNF) in this section. Orders with short duration will gain high priority to get dispatched first. Once there are multiple orders with the same estimated trip time, then orders with higher prices will be served first.

• •

REV: The revenue-based algorithm focuses on a higher ADI, which corresponds to Accumulated on-Service Time (AST) in this section, by assigning vehicles to high price orders. Following the similar principle as described above, the price and duration of orders will be considered as primary and secondary factors respectively.

• •

CoRide+: To distinguish CoRide running in different environment: single order dispatching, and joint order dispatching and fleet management, we sign the former one CoRide and latter one CoRide+.

In order to analyze these performances in a more straightforward way, we mainly employ rule-based methods here. Also, we introduce AST calculated as the accumulated on-service time and TNF computed as the total number of finished orders as the new metrics corresponding to ADI and ORR respectively. In order to further analyze these performances in a long-term way, we select one vehicle starting at grid 12, trace 10 timesteps and record its trajectory (as Figure 7 shows), then conclude these results in Table 2. Although we only record the first 10 timesteps, we can observe that our proposed methods, both CoRide+ and CoRide, are guiding the vehicle to regions with larger entropy. This is benefit from architecture where the state of both manager and worker, and ranking feature $e_{i}$ take the grid information into consideration. In contrast, other methods greedily optimize either AST (ADI) or TNF (ORR) and ignore these information. After taking a close look at Table 2, we can find that CoRide and CoRide+ share the same trajectory on discounted rate 20% and differ greatly when discounted rate moves to 30% and 40 %. This can be explained by regarding CoRide+ as a combined design between our proposed model CoRide and joint order dispatching and fleet management setting. Namely, CoRide is actually a special case of CoRide+, where fleet management is unable. Equipped with fleet management, CoRide+ allows the vehicle move to and serve order in the hotspots more directly than CoRide. Also, when discounted rate varies from 20% to 40%, fleet management enables CoRide+ with better adaptation and achieve stable performance, even can ignore the dynamics of order distributions in some cases.

According to aforementioned analysis, we can conclude that (i) CoRide+ achieves not only the state-of-the-art but a more stable result benefiting from joint order dispatching and fleet management setting; (ii) both CoRide and CoRide+ can direct the vehicle to grids with larger entropy via taking grid information into consideration.

7. Conclusion and future work

In this paper, we proposed CoRide, a hierarchical multi-agent reinforcement learning solution to combine order dispatching and fleet management for multi-scale ride-hailing platforms. The results on multi-city real-world data as well as analytic synthetic data show that our proposed algorithm achieves (i) a higher ADI and ORR than aforementioned methods, (ii) a multi-scale decision-making process, (iii) a hierarchical multi-agent architecture in the ride-hailing task and (iv) a more stable method at different cases. Note that CoRide could achieve fully decentralized execution and incorporate closely with other geographical information based model like estimating time of arrival (ETA) (Wang et al., 2018a) theoretically. Thus, it’s interesting to conduct further evaluation and investigation. Also, we notice that applying hierarchical reinforcement learning in real-world scenarios is very challenge and our work is just a start. There is much work for future research to improve both stability and performance of hierarchical reinforcement learning methods on real-world tasks.

Acknowledgments. The corresponding author Weinan Zhang thanks the support of National Natural Science Foundation of China (Grant No. 61702327, 61772333, 61632017). We would also like to thank our colleague in DiDi for constant support and encouragement.