The Green Choice: Learning and Influencing Human Decisions on Shared Roads

TL;DR

This paper proposes a method to influence human driving decisions and autonomous ride choices on shared roads to reduce congestion and improve traffic flow, using a preference learning algorithm and pricing strategies.

Contribution

It introduces a novel model for autonomous ride choice behavior, combined with a preference learning algorithm and an optimization framework for traffic management.

Findings

Pricing and routing strategies reduce congestion effectively.

Preference learning improves decision influence accuracy.

Optimized schemes maximize road usage and minimize delays.

Abstract

Autonomous vehicles have the potential to increase the capacity of roads via platooning, even when human drivers and autonomous vehicles share roads. However, when users of a road network choose their routes selfishly, the resulting traffic configuration may be very inefficient. Because of this, we consider how to influence human decisions so as to decrease congestion on these roads. We consider a network of parallel roads with two modes of transportation: (i) human drivers who will choose the quickest route available to them, and (ii) ride hailing service which provides an array of autonomous vehicle ride options, each with different prices, to users. In this work, we seek to design these prices so that when autonomous service users choose from these options and human drivers selfishly choose their resulting routes, road usage is maximized and transit delay is minimized. To do so, we formalize a model of how autonomous service users make choices between routes with different price/delay values. Developing a preference-based algorithm to learn the preferences of the users, and using a vehicle flow model related to the Fundamental Diagram of Traffic, we formulate a planning optimization to maximize a social objective and demonstrate the benefit of the proposed routing and learning scheme.

Figures

Tables

Table 1. TABLE I: Results of Routing Simulation

$\theta$	Avg. Latency (seconds)	Flow (cars/second)
$1$	$90.41$	$0.4412$
$20$	$97.03$	$1.2746$
${10}^6$	$111.28$	$1.5964$

Table 2. TABLE II: Results of Real-User Experiments

	Anticipated			Actual			BNE1		BNE2
	Avg. Latency	Flow	Profit	Avg. Latency	Flow	Profit	Avg. Latency	Flow	Avg. Latency	Flow
Case 1	$338.10$	$1.6622$	$11.11$	$283.30$	$1.0429$	$3.85$	$501.97$	$1.0429$	Infeasible
Case 2	$510.97$	$1.5592$	$4.50$	$537.30$	$1.5048$	$3.95$	$562.20$	$1.5048$	$562.20$	$1.6000$
Case 3	$921.06$	$1.2677$	$4.59$	$957.71$	$1.3000$	$3.99$	$1756.89$	$1.3000$	$1756.89$	$1.3000$
Case 4	$2083.42$	$1.2383$	$48.48$	$2089.53$	$1.2048$	$46.32$	$4194.26$	$1.2048$	$4194.26$	$1.3000$
Case 5	$2576.36$	$1.5538$	$40.01$	$2579.52$	$1.6000$	$49.33$	$2720.00$	$1.6000$	$2720.00$	$1.6000$

Equations

$${\tilde{n}}_i({\alpha }_i):=\frac{b_i}{{\alpha }_i{h}_i^{\text{a}}+(1-{\alpha }_i)h_i^{\text{h}}},$$

$$s_i=\{\begin{array}{ll}0& n_i^{\text{h}}+n_i^{\text{a}}\le {\tilde{n}}_i(n_i^{\text{a}}/(n_i^{\text{h}}+n_i^{\text{a}}))\\ 1& \text{otherwise}.\end{array}$$

$$f_i(n_i^{\text{h}},n_i^{\text{a}}):=$$

$$\{\begin{array}{ll}{\bar{v}}_i\cdot (n_i^{\text{h}}+n_i^{\text{a}}),& \text{if }{n}_i^{\text{h}}+n_i^{\text{a}}\le {\tilde{n}}_i({\alpha }_i)\\ \frac{{\bar{v}}_i\cdot {\tilde{n}}_i({\alpha }_i)\cdot ({\bar{n}}_i-(n_i^{\text{h}}+n_i^{\text{a}}))}{{\bar{n}}_i-{\tilde{n}}_i({\alpha }_i)},& \text{if }{\tilde{n}}_i({\alpha }_i)\le n_i^{\text{h}}+n_i^{\text{a}}\le {\bar{n}}_i\\ 0,& \text{otherwise }.\end{array}$$

$${\ell }_i(f_i^{\text{h}},f_i^{\text{a}},s_i)=\{\begin{array}{ll}\frac{d_i}{{\bar{v}}_i}& \text{if }{s}_i=0\\ d_i\left(\frac{{\bar{n}}_i}{f_i^{\text{h}}+f_i^{\text{a}}}+\frac{{\tilde{n}}_i({\alpha }_i)-{\bar{n}}_i}{{\bar{v}}_i\cdot {\tilde{n}}_i({\alpha }_i)}\right)& \text{if }{s}_i=1.\end{array}$$

$$f_i^{\text{h}}>0⟹{\ell }_i(f_i^{\text{h}},f_i^{\text{a}},s_i)\le {\ell }_{i^{\prime }}(f_{i^{\prime }}^{\text{h}},f_{i^{\prime }}^{\text{a}},s_{i^{\prime }})\forall i,i^{\prime }\in [N].$$

$$D=\{i\in [N]|(p_i>p_{i^{\prime }}\wedge {\ell }_i\ge {\ell }_{i^{\prime }})\vee (p_i\ge p_{i^{\prime }}\wedge {\ell }_i>{\ell }_{i^{\prime }})$$

$$\text{ for some }{i}^{\prime }\in [N]\}.$$

$$r_j(\mathbf{\ell },\mathbf{p},i)=$$

$$\{\begin{array}{ll}-{{\omega }_j}_1{\ell }_i-{{\omega }_j}_2{p}_i& \text{if user }j\text{ chooses road }i\text{ and }i\in \bar{D}\\ -\infty & \text{if user }j\text{ chooses road }i\text{ and }i\in D\\ -{\zeta }_j{\ell }^w& \text{if user }j\text{ declines the service ,}\end{array}$$

$$P(\text{user }j\text{ chooses route }i)=\frac{\exp \left(r_j(\mathbf{\ell },\mathbf{p},i)\right)}{{\sum }_{i^{\prime }\in [N]}\exp \left(r_j(\mathbf{\ell },\mathbf{p},i^{\prime })\right)}.$$

$$q_i(\mathbf{\ell },\mathbf{p})={\int }_0^{\infty }{\int }_0^{\infty }{\int }_0^{\infty }$$

$$g(\mathbf{\omega },\zeta )P(\text{user chooses route }i|\mathbf{\omega },\zeta )d{\omega }_{1}d{\omega }_{2}d\zeta .$$

$$P({\mathbf{\omega }}_j,{\zeta }_j|{\mathcal{D}}_j)$$

$$=P({\mathbf{\omega }}_j,{\zeta }_j)\prod _{m}P({{\mathcal{D}}_j}_m|{\mathbf{\omega }}_j,{\zeta }_j)$$

$${\text{query}}_m^{\ast }$$

$$\displaystyle\qquad P(\boldsymbol{\omega}_{j},\zeta_{j}|{\mathcal{D}_{j}}_{1:m-1})P({\mathcal{D}_{j}}_{m}|\boldsymbol{\omega}_{j},\zeta_{j},{\mathcal{D}_{j}}_{1:m-1})\Big{]}$$

$$=\underset{{\text{query}}_m}{\mathrm{argmin}}{\mathbb{E}}_{{{\mathcal{D}}_j}_m}\left[P({{\mathcal{D}}_j}_m|{\mathbf{\omega }}_j,{\zeta }_j,{{\mathcal{D}}_j}_{1:m-1})\right]$$

$${\text{query}}_m^{\ast }$$

$$P({\mathcal{D}{}_j}_m$$

$$=\sum _{{\mathbf{\omega }}_j,{\zeta }_j}P({\mathbf{\omega }}_j,{\zeta }_j|{\mathcal{D}{}_j}_{1:m-1})P({\mathcal{D}{}_j}_m|{{\mathcal{D}}_j}_{1:m-1},{\mathbf{\omega }}_j,{\zeta }_j)$$

$$\stackrel{\cdot }{=}\frac{1}{M}\sum _{{\bar{\mathbf{\omega }}}_j,{\bar{\zeta }}_j}P({{\mathcal{D}}_j}_m|{\bar{\mathbf{\omega }}}_j,{\bar{\zeta }}_j,{{\mathcal{D}}_j}_{1:m-1})$$

$${\text{query}}_m^{\ast }$$

$$\stackrel{\cdot }{=}\underset{{\text{query}}_m}{\mathrm{argmin}}\sum _{{\mathcal{D}{}_j}_m}P({\mathcal{D}{}_j}_m|{\mathcal{D}{}_j}_{1:m-1})\sum _{{\bar{\mathbf{\omega }}}_j,{\bar{\zeta }}_j}P({\mathcal{D}{}_j}_m|{\bar{\mathbf{\omega }}}_j,{\bar{\zeta }}_j,{\mathcal{D}{}_j}_{1:m-1})$$

$$\stackrel{\cdot }{=}\underset{{\text{query}}_m}{\mathrm{argmin}}\sum _{{{\mathcal{D}}_j}_m}{\left(\sum _{{\bar{\mathbf{\omega }}}_j,{\bar{\zeta }}_j}P({{\mathcal{D}}_j}_m|{\bar{\mathbf{\omega }}}_j,{\bar{\zeta }}_j)\right)}^2$$

$$J({\mathbf{f}}^{\text{h}},{\mathbf{f}}^{\text{a}},\mathbf{s})=\frac{{\sum }_{i\in [N]}(f_i^{\text{h}}+f_i^{\text{a}}){\ell }_i(f_i^{\text{h}},f_i^{\text{a}},s_i)}{{\sum }_{i\in [N]}(f_i^{\text{h}}+f_i^{\text{a}})}-\theta \sum _{i\in [N]}(f_i^{\text{h}}+f_i^{\text{a}}).$$

$$\underset{{\mathbf{f}}^{\text{h}},{\mathbf{f}}^{\text{a}},\mathbf{p}\in {\mathbb{R}}_{\ge 0}^N,k\in [N],s_{k:N}\in \{0,1{\}}^{N-k+1}}{\min }J({\mathbf{f}}^{\text{h}},{\mathbf{f}}^{\text{a}},\mathbf{s})$$

$$\sum _{i\in [N]}{f}_i^{\text{h}}={\lambda }^{\text{h}}$$

$$f_i^{\text{a}}={\lambda }^{\text{a}}{q}_i(\mathbf{\ell }({\mathbf{f}}^{\text{h}},{\mathbf{f}}^{\text{a}},\mathbf{s}),\mathbf{p}),\forall i\in [N]$$

$$a_k\le {\ell }_k(f_k^{\text{h}},f_k^{\text{a}},s_k)\le a_{k+1}$$

$$f_i^{\text{h}}=0,\forall i\in [N]\setminus [k]$$

$${\ell }_i(f_i^{\text{h}},f_i^{\text{a}},1)={\ell }_k(f_k^{\text{h}},f_k^{\text{a}},s_k),\forall i\in [k-1]$$

$$f_i^{\text{h}}+f_i^{\text{a}}\le \bar{F}\left(f_i^{\text{a}}/(f_i^{\text{a}}+f_i^{\text{h}})\right),\forall i\in [N]$$

$$\sum _{i\in [N]}\left(f_i^{\text{a}}{p}_i-f_i^{\text{a}}{d}_{i}c\right)\ge \bar{P}$$

$$q_i(\mathbf{\ell },\mathbf{p})\stackrel{\cdot }{=}\frac{1}{M}\sum _{\bar{\mathbf{\omega }},\bar{\zeta }}P(\text{user chooses route }i|\bar{\mathbf{\omega }},\bar{\zeta })$$

$$e_{x,y}=\Vert x/y-\hat{x}/\hat{y}{\Vert }_1$$

Full text

I Introduction

Road congestion is a major and growing source of inefficiency, costing drivers in the United States billions of dollars in wasted time and fuel [1]. The emergence of autonomous vehicles promises to improve the efficiency of road usage to some extent by forming platoons and smoothing traffic flow. However, if drivers make their routing decisions selfishly and seek to minimize their experienced delay, this can result in suboptimal network performance [2]. Moreover, in the presence of selfish users, the increase in road capacity induced by replacing some human-driven vehicles with autonomous ones can paradoxically worsen average transit user delay [3]. The adverse effects arise from the way humans make decisions about how to route themselves, and the resulting inefficiency can be very large, even unbounded [4].

It is therefore important to consider the effect that self-interested users will have on the system. Previous works have shown that if autonomous users are altruistic, average latency can be greatly reduced [4]. However, it’s not realistic to assume altruism in many scenarios; in this paper we consider how to monetarily incentivize such prosocial behavior. We consider the setting in which use of autonomous vehicles is offered by some benevolent ride hailing service or social planner which chooses prices for each of the routes. The users of the autonomous service will choose their routes, as well as whether or not they even want to travel, based on their time-money tradeoff. Then the remaining population of human drivers will choose routes that minimize their delay. The role of the social planner is then to choose prices that will optimize some social objective, which includes maximizing road usage and minimizing average delay. Our model can be viewed as an indirect Stackelberg game – a game in which a social planner controls some fraction of the population’s actions and the remainder of the population responds selfishly [5, 6]. However, in our model the planner can only control its portion of the vehicle flow via pricing.

In order to do so effectively, the social planner needs a model for how people make decisions between routes with various prices and latencies. We model these choices as being made based on a modification of Luce’s choice axiom [7, 8] and we model human drivers as selfish agents who reach a Wardrop Equilibrium, a configuration in which no one could decrease their travel time by switching paths [9].

Moreover, we use a method of actively learning the choices of autonomous service users via a series of queries [10, 11]. This enables the planner to predict how autonomous service users will react to a set of options after a relatively low number of training queries. We verify that this method accurately predicts human choice, and that our planning algorithm can use this to improve network performance, via experiment.

We summarize our contributions as follows.

• •

Learning Human Decisions: We develop an active preference-based algorithm to learn how people value the tradeoff between time and money, and choose their preferred transportation option. This enables learning a model of humans’ routing choices in a data-efficient manner.

• •

Influencing Human Decisions: We formulate and solve an optimization for ride hailing service to minimize congestion and maximize the road network utilization based on the learned model of human choice, while constraining a minimum profit for the service supplier.

• •

We validate the learning algorithm and the optimization formulation via simulations and user studies. Our results suggest carefully designed pricing schemes can provide significant improvements for traffic throughput.

**Related Work. **Previous works have shown the potential benefits that autonomous vehicles can have for traffic networks, by increasing road capacity through platooning [12, 13], damping shockwaves of slowing vehicles [14, 15, 16], managing merges [17], and decongesting freeways in the event of accidents [18]. In [3], the authors show that the capacity improvement from platooning can actually worsen the total delay experienced by users of the networks. Relatedly, many works (e.g. [19, 20], on shared roads, [21]) analyze and bound the inefficiency that can arise from network users choosing their routes selfishly instead of following some optimal routing.

As we use financial incentives to influence this behavior, our formulation is related to work in tolling [22], including works that consider users with different sensitivities to tolls [23, 24]. In contrast to many of these works, as well as some works that develop efficient algorithms for rebalancing an autonomous fleet [25], we consider a probabilistic model for human choice that has been empirically shown to reflect human choice making processes [26], and we model vehicle flow on shared roads based on the foundational Fundamental Diagram of Traffic (FDT) [27, 4]. This model captures the bifurcated nature of vehicle flow – when density is low, the road is in free-flow and flow increases with vehicle density, and when density is high, the road is congested and flow decreases as density increases.

On the human choice models, there have recently been a lot of effort on learning human reward functions which are assumed to be sufficient to model the preferences. Inverse reinforcement learning [28, 29, 30, 31] and preference-based learning [10, 11, 32, 33] are the most popular choices. In this paper, we employ preference-based learning as it is a natural fit to our problem. We actively synthesize queries as a non-trivial generalization and extension of [10] for data-efficiency and better usability.

II Problem Setting

**Vehicle Flow Model. **We consider a nonatomic model in which individual users control an infinitesimal unit of vehicle flow and this quantity is treated as a continuous variable. To model flow of mixed-autonomous traffic on roads, we use our model in [4], summarized in this section. This is an extension of the Fundamental Diagram of Traffic [27], in which flow inhabits one of two regimes – free flow, in which the flow on a road increases as vehicle density increases, and congestion, in which flow decreases as density increases. The flow decreases until the jam density, $\bar{n}_{i}$, at which point the flow ceases entirely. Similarly, the relationship between latency and flow on a road shows a similar divide: in free flow latency is constant with respect to flow, and in congestion latency increases as flow decreases. These behaviors are shown in Fig. 1. $\bar{v}_{i}$ denotes the free-flow speed of the road.

The transition between free flow and congestion occurs when the density on a road exceeds the critical density of that road, denoted $\tilde{n}_{i}$. We model the effect of autonomous vehicles as increasing this critical density, since autonomous cars can maintain shorter headways than human drivers, as follows.

[TABLE]

where $\alpha_{i}\in{[0,1{]}}$ is the autonomy level, or fraction of vehicles on road $i$ that are autonomous, $b_{i}$ is the number of lanes on that road, and $h^{\text{a}}_{i}$ and $h^{\text{h}}_{i}$ are the space occupied by an autonomous and human driven car, respectively, when traveling at the nominal velocity of the road. This model [21] assumes that either the autonomous vehicles maintain the same headway to any vehicle they may follow or that they can rearrange themselves on a road to form large platoons [34, 35], as shown in Fig. 2. Note that if autonomous vehicles can keep shorter headways than human drivers (i.e. $h^{\text{a}}_{i}<h^{\text{h}}_{i}$), the presence of autonomous cars increases the critical density and therefore also increases the maximum flow possible on a road, i.e. $\bar{F}_{i}(1)>\bar{F}_{i}(0)$ where $\bar{F}_{i}$ is a function of autonomy level that gives the maximum capacity of road $i$. Fig. 1 shows the effect of increased autonomy on a road.

We use $s_{i}$ to denote the state of road $i$ such that $s_{i}=0$ denotes it is in free flow and $s_{i}=1$ denotes it is congested. We use $n^{\text{h}}_{i}$ and $n^{\text{a}}_{i}$ to denote the density of human-driven and autonomous vehicles, respectively on road $i$. Accordingly,

[TABLE]

We model the road as first-in-first-out, so $\alpha_{i}=n^{\text{a}}_{i}/(n^{\text{h}}_{i}+n^{\text{a}}_{i})=f^{\text{a}}_{i}/(f^{\text{h}}_{i}+f^{\text{a}}_{i})$, where $f^{\text{h}}_{i}$ and $f^{\text{a}}_{i}$ respectively are the human-driven and autonomous vehicle flow on road $i$.

The flow on a road, $f_{i}=f^{\text{h}}_{i}+f^{\text{a}}_{i}$ is a function of the density of each vehicle type as follows.

[TABLE]

We can then write the latency as a function of vehicle flow and the state of the road [6, 4]:

[TABLE]

where $d_{i}$ is the length of road $i$.

We now know how to relate the vehicle flow to road latency so we can start reasoning about how people choose their routes.

**Problem Overview. **We describe the various facets of the problem, preceded by a high-level overview of the objective. We assume that the demand of human drivers is fixed, and that the demand of people using the autonomous service is elastic – if prices and latencies are high, some people may choose not to use the autonomous mobility service. Our goal is then to simultaneously maximize the number of autonomous service users that can use the road, and minimize the average delay experienced by all the people using the roads.

Our control variables in this optimization are the latencies on each road and the prices offered to the users for traveling on the routes. However, we can’t arbitrarily choose prices and latencies – we need to respect

the characteristics of the roads, in terms of how the flow demand for a road corresponds to the latency on that road, and 2. 2.

how people make decisions, making sure that the number of people who choose each option corresponds to the latencies of the roads described in the options.

Moreover, we want to be fair and must therefore offer the same pricing and routing options to all users of the autonomous service.

How a population of users chooses between a variety of price and latency pairs depends on their valuation of time and money. Without knowing this choice model we cannot plan vehicle flows and will not be able to ensure the resulting configuration matches our vehicle flow models for the roads. Also, since different populations may have different parameter distributions, we need to learn this tradeoff for our population before we can estimate how many people will choose which option. To untangle these constraints, in the subsequent sections we describe our models for vehicle flow and human choices. Following that we return with a mathematical formulation of this optimization.

III Learning the Human Choice Model

In this section we describe how both human drivers and the autonomous service users make routing decisions. We then provide a data-efficient learning algorithm for estimating the way that the autonomous service users make routing decisions, allowing us to predict the response a population will have to a set of latency/price options.

*Human Drivers. *We consider a network of $N$ parallel roads, an example of which is shown in Fig. 3. We assume that no two roads have the same free-flow latency and denote the free-flow latency of path $i$ by $a_{i}$. We order the indices such that $a_{1}<a_{2}<\ldots<a_{N}$, and we use $[k]$ to denote the set of the first $k$ roads.

For the human drivers, we assume their only consideration is minimizing their commute time. This leads to a Wardrop Equilibrium [36, 37], which, on parallel roads, means that if one road has positive flow on it, all other roads must have higher or equal latency. Formally,

[TABLE]

This implies that all selfish users experience the same latency. It is therefore useful to consider the flow-latency diagrams of roads when studying which equilibria exist – by fixing the latency on the y-axis, one can reason about which roads must be congested to achieve that equilibrium. As shown in Fig. 4, equilibria may have one road in free-flow and rest congested, or all may be congested, assuming no two roads have the same free-flow latency [6, 4].

*Autonomous Service Users. * Though human drivers are motivated directly only by delay, users of the autonomous service experience cost in both delay and the price of the ride. We model the users as having some underlying reward function, which is parameterized by their time/money tradeoff, as well as the desirability of the option of traveling by some other means such as walking or public transit. We assume that a strictly dominated option in terms of latency and prices is completely undesirable. Formally, this set of dominated roads is defined as

[TABLE]

We also define the complementary set, i.e., nondominated roads, $\bar{D}={[}N{]}\backslash D$. We model the reward function of user $j$ for choosing road $i$ as follows:

[TABLE]

where $\boldsymbol{\ell}$ denotes the vector of road latencies, $\boldsymbol{p}$ denotes prices, ${\boldsymbol{\omega}_{j}}=\begin{bmatrix}{\omega_{j}}_{1}&{\omega_{j}}_{2}\end{bmatrix}^{T}$ characterizes the users’ time/money tradeoff and $\zeta_{j}$ specifies their willingness to use an alternative option, which has delay $\ell^{w}$. This alternate option could be walking, biking, or public transportation; the nature of this option determines the delay as well and influences $\zeta_{j}$.

We don’t assume that users are simple reward maximizers. Rather, we draw from Luce’s Choice Axiom [7] to model the probability with which users choose each option, in combination with a softmax rule [26], formalized as follows.

[TABLE]

In order to determine optimal pricing, we want to know how many users will choose each route as a function of the route prices and latencies. To this end, we define $q_{i}(\boldsymbol{\ell},\boldsymbol{p})$ as the expected fraction of autonomous service users that will choose route $i$. If the parameter distribution for autonomous service users is $g(\boldsymbol{\omega},\zeta)$, then

[TABLE]

This expression relate prices and latency to human choices, enabling us to determine the prices that will maximize our social objective. This will be important in constraining our optimization to only consider latency/price options that correspond to the desired vehicle flows.

**Data-Efficient Learning of Human Reward Functions. ** While the routes in a specific network can be fully modeled with the physical properties and the speed limits, user’s decision models must be learned in a data-driven way. As described above, reward function parameters $(\boldsymbol{\omega},\zeta)$ are sufficient to model the decisions of a user.

The parameters will be different among the users. While a business executive might prefer paying extra to reach their company quickly, a graduate student may decide to go to the lab a little later in order to save a few dollars. Therefore, we have to learn personalized parameters $\boldsymbol{\omega}$ and $\zeta$. Although it might be possible to make a rough guess based on personal information, such as age, income, occupation, location, etc; there are a few problems about this approach: 1) people might be unwilling to share those information accurately, 2) the variance will be high, since the decisions are also affected by the personal traits that are hard to quantify, 3) we do not know how we will instantiate the parameters, because putting this as a regression problem would require some users to explicitly quantify their $\boldsymbol{\omega}$ and $\zeta$, which is rather unrealistic.

Due to these problems, we choose to learn the parameters from users’ previous choices, which is known as preference-based learning. If user $j$ chooses from a variety of options, the user’s choice gives us a noisy estimate of which road $i\in{[}N{]}$ maximizes their reward function $r_{j}(\boldsymbol{\ell},\mathbf{p},i)$. We could start from either uniform priors or priors informed by domain knowledge, then sample from the distribution $g(\boldsymbol{\omega},\zeta)$.

However, a major drawback of doing so is how quickly we learn the user preferences. Preference-based learning suffers from the small amount of information that each query carries. For example, if we show $4$ options to a user (including the option to decline the service), then the maximum information we can get from that query is just $2$ bits. In fact, the information gained can be much lower in later queries, because the posterior is already a better estimate than the original prior distribution. To tackle this issue, previous works pose the query generation/synthesis problem as a submodular optimization and maximize a measure of the expected change in the learned distribution after the queries. [10, 11].

While those works focused only on pairwise queries, in this case we expect to pose several route options to the users and therefore need more general query types. By using these more general queries that offer a variety of routes with varied latencies and prices, we can consider a number of ways of using this learning framework to learn the human preference distribution.

• •

We could do a user study on a few people to learn a good prior distribution. While we could achieve this using random queries, it would take too many queries to learn a useful distribution and most of the queries would not be sensible. By utilizing active learning, we are able to learn the parameters of each user using only a few queries.

• •

We could do an exploration/exploitation strategy if we are allowed to break the fairness constraint a few times for some small portion of the users. This could be possible through user-specific campaigns. That is, for each user we may either choose to use the learned model or to offer special rates that would help us profile the user better.

• •

We could do an initial profiling study for each new user, where we ask them queries synthesized by the active learning framework.

To implement any of these options, we formulate the following active learning optimization. First, we discuss the general preference-based learning framework. Given the data from previous choices of user $j$, which we denote as $\mathcal{D}_{j}$, we can formalize the probability of $(\boldsymbol{\omega}_{j},\zeta_{j})$ being true parameters for that user as follows:

[TABLE]

where ${\mathcal{D}_{j}}_{m}$ denotes the route user $j$ chose in their $m^{\textrm{th}}$ choice (with ${\mathcal{D}_{j}}_{m}=0$ meaning that the user saw the options, but preferred the alternative option). The first expression is due to Bayes’ rule and the equality is due to the assumption that the users’ choices are conditionally independent from each other given the reward function parameters.

We can readily obtain the second term from the human choice model. For the prior, we can simply use a uniform distribution over nonnegative values of the parameters. The prior can be crucial especially when we do not have enough data for a new user. In such settings, we incorporate domain knowledge to start with a better prior than a uniform distribution.

We can then use this unnormalized $P(\boldsymbol{\omega}_{j},\zeta_{j}|\mathcal{D}_{j})$ to obtain the samples of $(\boldsymbol{\omega}_{j},\zeta_{j})$ using Metropolis-Hastings algorithm. Doing this for each user, which can be easily parallelized, we can directly obtain the samples $(\boldsymbol{\bar{\omega}},\bar{\zeta})\sim g(\boldsymbol{\omega},\zeta)$.

Next we formulate the active learning framework, which is needed so that it will not take an excessive number of queries to learn human preferences. For this, we want to maximize the expectation of the difference between the prior and the unnormalized posterior:

[TABLE]

As we will use the sampled $(\boldsymbol{\omega}_{j},\zeta_{j})$ for computing the probabilities of each route choice for the expectation over ${\mathcal{D}_{j}}_{m}$, we can write the optimization as:

[TABLE]

where we have $M$ samples denoted as $(\boldsymbol{\bar{\omega}}_{j},\bar{\zeta}_{j})$, $\stackrel{{\scriptstyle\boldsymbol{\cdot}}}{{=}}$ denotes asymptotic equality as $M\to\infty$, and the term $1/M$ is canceled. Using the law of total probability, we can also write

[TABLE]

which then leads to

[TABLE]

We can easily compute this objective value for any given $\textrm{query}_{m}$. This optimization is unfortunately nonconvex due to the human choice model. As in previous works, we assume local optima is good enough [10, 11]. We then use a Quasi-Newton method (L-BFGS [38]) to find the local optima, and we repeat this for $1000$ times starting from random initial points so that we can get closer to the global optimum.

IV Influencing Human Decision Making

**Problem Formulation. ** We are now ready to formulate our objective. Since the demand of autonomous service users is elastic, we cannot just minimize the average latency. That would result in extremely high prices to reduce latency by keeping autonomous service users off the network. Hence, we consider the objective to be a combination of maximizing road usage and minimizing average travel time. We parameterize this tradeoff with parameter $\theta\geq 0$ in the cost function

[TABLE]

Given $\mathbf{q}(\boldsymbol{\ell},\boldsymbol{p})$, the cost function $J(\boldsymbol{f}^{\text{h}},\boldsymbol{f}^{\text{a}},\boldsymbol{s})$, inelastic flow demand of human drivers $\lambda^{\text{h}}$, and elastic demand of autonomous users $\lambda^{\text{a}}$, we are ready to formulate the planning optimization. The most straightforward way is to optimize jointly over $\boldsymbol{f}^{\text{h}},\boldsymbol{f}^{\text{a}},\boldsymbol{\ell}$ and $\boldsymbol{p}$. However, $\boldsymbol{\ell}$ is fully defined by $\boldsymbol{f}^{\text{h}},\boldsymbol{f}^{\text{a}}$ and $\boldsymbol{s}$. Hence, instead of $\boldsymbol{\ell}$, we use $\boldsymbol{s}$, which will help as the optimization is nonconvex and we can search over the possible values of $\boldsymbol{s}$. The problem is then formulated as:

[TABLE]

[TABLE]

with $s_{1:k-1}=1$ due to selfishness of human-driven vehicles, where $k$ is the longest road with human-driven vehicle flow, $\bar{P}$ is the minimum profit per unit time that we want to make from autonomous service users and $c$ is the constant fuel cost per unit length, which can be easily calculated by multiplying the vehicles’ fuel consumption per unit distance and the fuel price. We can describe the constraints as follows.

The human-driven vehicle flow demand is fulfilled. 2. 6.

Autonomous flow will be distributed based on the choice model described in the preceding section. 3. 7.

The “longest equilibrium road” has latency on the given interval of free flow latencies. 4. 8.

Human-driven cars are selfish, i.e. no human-driven car will experience higher latencies than the road $k$. 5. 9.

The congested roads have the same latency as the “longest equilibrium road”. 6. 10.

The maximum capacities of the roads are respected. 7. 11.

The minimum profit per unit time is satisfied.

We can further improve the search space by relying on the heuristic that the roads that are not used by the human-driven vehicles will be in free-flow, i.e. $s_{k+1:N}=0$. While we do not have a proof for this, we also note constructing counterexamples seems to require extremely careful design, which suggests the heuristic holds in general. Furthermore, the following proposition shows we could also set $s_{k}=0$ under an additional assumption.

Proposition 1

Under the assumptions that the reward function is homogeneous among the users and $\omega_{2}>0$, there exists a free-flow road $k$ in the optimal solution to the problem such that $\ell_{i}=\ell_{k}$ for $\forall i\in[k]$, and $f^{\text{h}}_{i}=0$ for $\forall i\in{[}N{]}\setminus[k]$ as long as the optimization is feasible.

The proof is given in Appendix.

Generalizations. We assumed all autonomous cars are controlled by a centralized social planner. To extend our framework to scenarios where this is not the case and the social planner has the control over a fraction of autonomous cars, we can simply do the following modifications: The optimization will also be over $\boldsymbol{f}^{\text{b}}\in\mathbb{R}_{\geq 0}^{N}$, which will now represent the autonomous flow that does not belong to the planner. We add the corresponding constraint of 5 for $f^{\text{b}}_{i}$. Similar to 8, we will also have $f^{\text{b}}_{i}=0$ for $i\in{[}N{]}\setminus{[}k{]}$ due to selfishness. We will also replace $f^{\text{a}}$’s in 7, 9 and 10 with $f^{\text{a}}+f^{\text{b}}$ with appropriate subscripts. These simple modifications enable a more general use.

**Solving the Optimization. ** After learning the distribution $g(\boldsymbol{\omega},\zeta)$, we first take $M$ samples $(\boldsymbol{\bar{\omega}},\bar{\zeta})\sim g(\boldsymbol{\omega},\zeta)$ as we previously described in Section III. Using these samples, we approximate the expected fraction of autonomous users that will choose route $i$ as:

[TABLE]

We then locally solve the nonconvex planning optimization using interior point algorithms [39, 40, 41]. We execute the algorithms with $100$ random initial points for each run to get closer to global optimum.

V Experiments & Results

To validate our framework, we conducted different simulations and a user study.

Hypotheses. We test three hypotheses that together suggest our framework successfully reduces traffic congestion through pricing, after it effectively learns the humans’ choice models:

H1: Our active learning algorithm is able to learn the presented human choice model of autonomous service users in a data-efficient way.

H2: Our planning optimization reduces the latencies by creating altruistic behavior through pricing.

H3: When used by humans, the overall framework works well and is advantageous over non-altruistic algorithms.

Implementation Details. In the planning optimization, we used the heuristic that $s_{k+1:N}=0$, but did not set $s_{k}=0$. We assumed there is only one alternative option for autonomous service users, and it is walking to the destination. We set $c=6\times 10^{-5}$ USD/meter, $\mathbf{b}=1$. We assumed the human-driven cars keep a $2$-second headway distance with the leading car, whereas autonomous cars keep $1$-second distance. We set the length of the cars as $5$ meters, and the minimum gap between two cars as $2$ meters.

Experiments and Analyses. To test H1, we simulated $5$ autonomous service users with different preferences. We tested our data-efficient learning framework by asking two sets of $200$ queries, each of which consisted of $4$ road options, similar to Fig. 7, and a walking option, to the simulated users. The queries were generated actively in the first set and randomly in the second. After each query, we recorded the sample $(\bar{\boldsymbol{\omega}},\bar{\zeta})$ which has the highest likelihood as our estimates.

Fig. 5 shows how the estimates evolved within active learning setting for one of the users. All values are overestimated initially. This is intuitively because getting noiseless responses has higher likelihood. As we query more, accepting some of the responses as noisy maximizes the likelihood. Therefore, the values start decreasing.

Another important observation is that the estimates of the parameters increase and decrease together even in the early iterations. This suggests we are able to learn the ratio between the parameters, e.g. $\omega_{1}/\omega_{2}$ very quickly. To check this claim, we used the following error metric:

[TABLE]

where $x,y\in\{\omega_{1},\omega_{2},\zeta\}$ and $\hat{x},\hat{y}$ stand for the parameters $x,y$ of the sample with the highest likelihood. Figure 6 shows how this error decreases with increasing number of queries. It also shows how active querying enables data-efficient learning compared to the random querying baseline.

This indicates we are able to learn the relationship between parameters even under $20$ queries with active learning. All these results strongly support H1.

The fact that we are able to learn the ratios implies we can estimate which road the user is most likely to choose. We will only be unsure about how noisy the user is if the parameter estimates did not converge yet. Therefore, we can still use the estimates for our planning optimization formulation even when we have small number of queries.

To test H2, we adopt $3$ of the equilibria from [4] for comparison, namely, Nash Equilibrium where all drivers are selfish; Best Nash Equilibrium which gives the smallest average latency again with selfish drivers; and Fully Altruistic Best Nash Equilibrium which gives the smallest average latency where autonomous users are fully altruistic and human-driven cars route selfishly. Here, we give the average latencies for the specific $4$-road network from that study which we visualize in Fig. 7, and where $\lambda^{\text{h}}=0.4,\lambda^{\text{a}}=1.2$ cars per second:

• •

Nash Equilibrium: $400.00$ seconds

• •

Best Nash Equilibrium: $125.66$ seconds

• •

Fully Altruistic Best Nash Equilibrium: $102.85$ seconds

We then assumed we perfectly learned the preferences of the $5$ simulated users. We ran the planning optimization with $\bar{P}=0$ and $3$ different $\theta$ to show the trade-off. The results are summarized in Table I.

It can be seen we can adjust the trade-off between average latency and the served flow by tuning $\theta$. Also, given the human preferences, even when we served (almost) all of autonomous demand, our framework outperforms Best Nash Equilibrium. This shows the effectiveness of our framework on creating altruism and supports H2.

For H3, we recruited $21$ subjects ($9$ female, $12$ male) with an age range from 19 to 60. To increase the diversity among user preferences, the subjects are selected from a diverse set of locations: California / Georgia / Illinois, USA; Ankara / Denizli, Turkey; Sao Paulo, Brazil; Singapore.

In the first phase of the experiment, each participant was asked $40$ queries ($4$ roads + $1$ walking option) which are all actively synthesized. No user has selected a strictly dominated option in any query, which indicates the data are reliable. We then used their responses to get the maximum likelihood estimates which we approximate as the sample $(\boldsymbol{\bar{\omega}},\bar{\zeta})$ that has the highest likelihood. Afterwards, we designed $5$ different road networks each with $4$ different roads and an additional route where people may choose to walk. The roads on the $5$ different networks cover a range of different road lengths from $1.8$ kilometers to $78$ kilometers. For each network, we also set different $\theta$, $\lambda^{\text{h}}$, $\lambda^{\text{a}}$, and $\bar{P}$. By assuming all autonomous flow is in the service of these $21$ subjects, or of the groups that match with their preferences, we locally solved the planning optimization to obtain the pricing scheme for each traffic network. We refer to the results of this optimization as anticipated values, e.g. anticipated profit.

In the second phase of the user study, we presented the route-price pairs and the walking option to the same $21$ subjects. For each of the $5$ networks, they picked the option they would prefer. Using these responses, we allocated the autonomous flow into the roads. However, it is technically possible that more users select a road than its maximum flow. To handle such cases, we assumed extra flow is transferred to the roads with smaller latencies without making the users pay more. If that is not feasible, extra flow is transferred to the slower roads, without any discount. While these break the fairness constraint, it rarely happens and affects only a very small portion of the flow. After autonomous flows are allocated, human driven cars selfishly chose their routes in a way to minimize the overall average latency. We refer to the results of this allocation as actual values, e.g. actual profit.

Table II compares the anticipated and the actual values. We report latencies in seconds, flows in cars per second, and profit is a rate value with the unit USD per second. In order to show how our framework incentivizes altruistic behavior, we also added two other selfish best Nash equilibrium baselines: one where the same flow as actual flow is routed (BNE1) and one where all flow demand is routed (no walking option) again under best Nash equilibrium (BNE2).

It can be seen that there is generally an alignment between anticipated and actual values. While the mismatch may be further reduced by doing more queries or having more users, the difference with BNE methods is significant. In all cases, our framework achieved to incentivize altruism, which yielded lower average latencies compared to BNE1. Especially in Case 3 and Case 5, our framework approximately halved the average latency compared to the best Nash equilibria. We visualize this difference in Fig. 8. Furthermore, our framework successfully reduced flow demand when satisfying the full demand is impossible (Case 1).

One caveat is the small amount of actual flow in Case 1, which also caused an important profit loss. This is because the roads are relatively shorter, and most users preferred walking over paying for an autonomous car. Our framework could not predict this, because the learned reward functions failed to accurately estimate the probabilities. This issue will be further discussed in Section VI.

VI Conclusion

In this work, we develop a method improving the efficiency of traffic networks shared between human-driven and autonomous vehicles by influencing the routing behavior of humans. Considering an autonomous service that offers a choice of a number of routes, we formulate an optimization to find the prices that influence autonomous user choices so that when users choose their routes, and the human-driven traffic routes solely based on travel delay, travel delay is minimized and road usage is maximized. In order to do so, we model how people choose between between routes with differing prices and latencies, and develop a method to actively learn the choice model parameters for a population of users. We verify these mechanisms through an experiment with real human choice data.

There are areas in which our mechanism can be improved. Specifically, one could further refine the reward function model for autonomous service users and not necessarily assume that it’s linear. One could also consider the human-driven vehicle demand to be elastic, meaning that people may choose to stay home or choose public transportation if congestion is high. Further, we find that when solving the planning optimization using actual data from user choices, the profit constraint is often unsatisfied. To manage this, we could instead include profit as part of the objective function or optimize for the worst-case performance.

We considered only a single alternative to the autonomous service use. However, a population may choose from a variety of options such as walking, biking, or taking the bus, $\zeta$ will have a multimodal distribution; we then need to learn the mixture. Finally, it is unclear if it is better to use many samples from our posterior on $(\boldsymbol{\omega},\zeta)$ or just the maximum likelihood sample while computing $\mathbf{q}$ for the planning optimization.

Future work can address these issues or expand in broader directions. It would be worthwhile to look at more general network topologies. Further, the information that people have access plays a huge role in decision making. It would be enlightening to develop choice models that incorporate this information availability as well as possibly other aspects of decision making, such as risk aversion. These expansions would do much to address congestion on roads that are soon to be shared between human drivers and autonomous vehicles.

Appendix

VI-A Proof of Proposition 1

Let us call the reward function that is homogeneous among the users as $\bar{r}$. Assume the optimal solution is $(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},{\boldsymbol{f}^{\text{h}}}^{*},{\boldsymbol{f}^{\text{a}}}^{*})$, and $\boldsymbol{\ell}^{*}_{k}>a_{k}$, where $k=\operatornamewithlimits{arg\,max}_{i}a_{i}$ subject to ${f^{\text{h}}}^{*}_{i}>0$. Due to selfishness, we know $\ell_{i}=\ell_{k}$ for $\forall i\in[k]$. Also, let $D^{*}$ (and $\bar{D}^{*}$) be the set of dominated (undominated) options under the optimal solution.

Let $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})$ be such that

• •

$\ell^{\prime}_{i}=\ell^{*}_{i}$ and $p^{\prime}_{i}=p^{*}_{i}$, for $\forall i\in{[}N{]}\setminus[k]$,

• •

$a_{k}\leq\ell^{\prime}_{k}<\ell^{*}_{k}$,

• •

$\ell^{\prime}_{i}=\ell^{\prime}_{k}$, for $\forall i\in[k-1]$,

• •

$p^{\prime}_{i}=p^{*}_{i}+\epsilon$ where $\epsilon>0$ for $\forall i\in[k]\cup D^{*}$,

• •

$\mathbf{q}(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})=\mathbf{q}(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*})$ for $i\in[k]\;.$

We also define $D^{\prime}$ and $\bar{D}^{\prime}$ similarly. The only two free variables of $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})$ are $\ell^{\prime}_{k}$ and $\epsilon$. To show such an $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})$ with the last constraint exists, we prove

1. For any $\ell^{\prime}_{k}$, there exists an $\epsilon$ such that $\bar{r}(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},i)=\bar{r}(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime},i)$ for $\forall i\in[k]$, 2) $D^{*}=D^{\prime}$. These together imply $\bar{r}(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},i)=\bar{r}(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime},i)$ for $\forall i\in{[}N{]}$, which then implies the last constraint.

The first one is easy to show. $\bar{r}(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},i)=\bar{r}(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},k)$ for $\forall i\in[k]$. For any $\ell^{\prime}_{k}$, we can set $\epsilon=\frac{\omega_{1}}{\omega_{2}}(\ell^{*}_{k}-\ell^{\prime}_{k})$ to get $\bar{r}(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime},i)=\bar{r}(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},i)$ for $\forall i\in[k]$ where $\boldsymbol{\omega}$ is associated with $\bar{r}$. The equality also holds for $i\in{[}N{]}\setminus[k]$, because the prices and latencies are the same between $(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*})$ and $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})$ in those roads.

We show $D^{*}=D^{\prime}$ in two steps:

1. $D^{*}\cap[k]=D^{\prime}\cap[k]$, since the roads had the same latency drop and price increase, and they cannot be dominated by any of ${[}N{]}\setminus[k]$, because they have strictly lower latency.

2. $D^{*}\setminus[k]=D^{\prime}\setminus[k]$, because the roads have the same latency between two configurations and the prices went up only for those in $D^{*}$, so they cannot dominate each other in a different way between the configurations. They also cannot dominate any of the roads in $[k]$, because they have strictly higher latency. And $i\in D^{\prime}\setminus[k]\implies i\in D^{*}\setminus[k]$, because the roads in $[k]$ had already lower latency in the optimal solution and their prices only went up.

Having showed $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})$ exists, we note $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime},{\boldsymbol{f}^{\text{h}}}^{\prime},{\boldsymbol{f}^{\text{a}}}^{*})$ is a feasible solution for some ${\boldsymbol{f}^{\text{h}}}^{\prime}$, because the distribution of autonomous service users remained the same and the prices went only up, which together means the profit constraint (11) is still satisfied. The other constraints are trivially satisfied by the construction of $(\boldsymbol{\ell}^{\prime},\boldsymbol{p}^{\prime})$ and as the optimal solution has to be feasible. Then, since the demand of human-driven cars are inelastic, the total served flow is the same between two solutions. However, the latencies of the roads in $[k]$ decreased, which means their capacities increased. Hence, $k\!\geq\!\operatornamewithlimits{arg\,max}_{i}a_{i}$ subject to ${f^{\text{h}}}^{\prime}_{i}\!>\!0$. Then, the average latencies of human-driven cars decreased. Similarly, the average latencies of autonomous service users may only go down. Hence, $J({\boldsymbol{f}^{\text{h}}}^{*},{\boldsymbol{f}^{\text{a}}}^{*},\boldsymbol{s}^{*})\!>\!J({\boldsymbol{f}^{\text{h}}}^{\prime},{\boldsymbol{f}^{\text{a}}}^{\prime},\boldsymbol{s}^{\prime})$, and $(\boldsymbol{\ell}^{*},\boldsymbol{p}^{*},{\boldsymbol{f}^{\text{h}}}^{*},{\boldsymbol{f}^{\text{a}}}^{*})$ cannot be the optimal solution.$\qquad\square$