Reward-Based Deception with Cognitive Bias

TL;DR

This paper proposes a novel framework for deception in adversarial interactions that exploits human cognitive biases, specifically prospect theory, to optimize resource allocation against bounded rational adversaries.

Contribution

It introduces a new approach to deception that leverages human cognitive biases and formulates resource allocation as a signomial program within a Markov decision process.

Findings

Effective resource allocation reduces defender's costs.

Simulation with real-world data demonstrates the framework's practicality.

Exploiting cognitive biases enhances deception strategies.

Abstract

Deception plays a key role in adversarial or strategic interactions for the purpose of self-defence and survival. This paper introduces a general framework and solution to address deception. Most existing approaches for deception consider obfuscating crucial information to rational adversaries with abundant memory and computation resources. In this paper, we consider deceiving adversaries with bounded rationality and in terms of expected rewards. This problem is commonly encountered in many applications especially involving human adversaries. Leveraging the cognitive bias of humans in reward evaluation under stochastic outcomes, we introduce a framework to optimally assign resources of a limited quantity to optimally defend against human adversaries. Modeling such cognitive biases follows the so-called prospect theory from behavioral psychology literature. Then we formulate the resource allocation problem as a signomial program to minimize the defender's cost in an environment modeled as a Markov decision process. We use police patrol hour assignment as an illustrative example and provide detailed simulation results based on real-world data.

Figures

Equations

$$\text{BODY}$$

$$f=c\cdot x_1^{a_1}\cdots x_n^{a_n},$$

$$g=\sum _{k=1}^K{c}_k\cdot x_1^{a_{1k}}\cdots x_n^{a_{nk}}.$$

$$f$$

$$g_i\le 1,i=1,\dots ,m,$$

$$h_j=1,j=1,\dots ,m,$$

$$P_{\pi }(\omega )=\nu (s_0)\prod _i\pi (s_i,a_i)T(s_i,a_i,s_{i+1}).$$

$$R(s)=g(U(s))\in {\mathbb{R}}^{+},$$

$$R_h(s)=f(U(s))\in {\mathbb{R}}^{+},$$

$$Q_t(s)=R(s)+\sum _a\sum _{s^{\prime }}\pi (s,a)T(s,a,s^{\prime })Q_{t+1}(s^{\prime }),$$

$$Q=\sum \nu (s)Q_0(s),$$

$$P({♢}^{\le H}{S}_s)=\sum _{\omega \in \Omega }P(\omega ).$$

$$P({♢}^{\le H}{S}_s)\le \lambda .$$

$$V(X)=\sum _{x\in O}v(x)w(p(x)),$$

$$r_a(s)=\sum _{s^{\prime }}R(s^{\prime })T(s,a,s^{\prime }).$$

$$p(x_{s^{\prime }})=T(s,a,s^{\prime }),\forall x_{s^{\prime }}\in O_{s,a}.$$

$$v(x_{s^{\prime }})=R_h(s^{\prime })=f(U(s^{\prime })).$$

$$\begin{array}{rl}{r}_a^h(s)& =\sum _{x_{s^{\prime }}\in O_{s,a}}v(x_{s^{\prime }})w(p(x_{s^{\prime }}))\\ & =\sum _{s^{\prime }}f(U(s^{\prime }))w(T(s,a,s^{\prime })).\end{array}$$

$$w(p)=\frac{p^{\gamma }}{(p^{\gamma }+(1-p{)}^{\gamma }{)}^{\frac{1}{\gamma }}},\gamma >0.$$

$$\pi (s,a)=\frac{r_a^h(s)}{{\sum }_{a^{\prime }}{r}_{a^{\prime }}^h(s)},$$

$$\text{minimize}Q=\sum \nu (s)Q_0(s)$$

$$\forall s\in S,t\in \{0,\dots ,H-1\},$$

$$Q_t(s)\ge R(s)+\sum _{a\in A}\sum _{s^{\prime }\in S}\pi (s,a)T(s,a,s^{\prime })Q_{t+1}(s^{\prime })$$

$$\forall s\in S,t\in \{0,\dots ,H-1\},$$

$$P_t(s)\ge \sum _{a\in A}\sum _{s^{\prime }\in S}\pi (s,a)T(s,a,s^{\prime })P_{t+1}(s^{\prime })$$

$$\forall t\in \{0,\dots ,H\},\forall s\in S_s,P_t(s)=1$$

$$\forall t\in \{0,\dots ,H\},\sum _{s\in S}\nu (s)P_t(s)\le \lambda$$

$$\forall s\in S,a\in A,$$

$$\pi (s,a)\sum _{a^{\prime }\in A}\sum _{s^{\prime }\in S}f(U(s^{\prime }))w(T(s,a^{\prime },s^{\prime }))$$

$$=\sum _{s^{\prime }\in S}f(U(s^{\prime }))w(T(s,a,s^{\prime }))$$

$$\forall s\in S,R(s)=g(U(s))$$

$$\sum _{s\in S}U(s)=D,$$

$$\displaystyle\forall i.1\leq i\leq n\quad\hat{f}=f[\hat{x}]\prod_{i=1}^{n}\Bigg{(}\dfrac{x_{i}}{\hat{x}(x_{i})}\Bigg{)}^{a_{i}},$$

$$\text{where }{a}_i=\frac{\hat{x}(x_i)}{f[\hat{x}]}\frac{\partial f}{\partial x_i}[\hat{x}].$$

$$\forall t\in \{0,\dots ,H\},\sum _{s\in S}\nu (s)P_t(s)\le \lambda \cdot \tau$$

$$\tau \ge 1.$$

$$\text{minimize}Q+\delta \cdot \tau$$

$$R(s)=\frac{C(s)}{U(s)}.$$

Full text

I Introduction

Deception refers to a deliberate attempt to mislead or confuse adversaries so that they may take strategies that are in the defender’s favor [1]. Deception can limit the effectiveness of an adversary’s attack, waste adversary’s resources and prevent the leakage of critical information [2]. It is a widely observed behavior in nature for self-defence and survival. Deception also plays a key role in many aspects of human society, such as economics [3], warfare [4], game [5], cyber security [2] and so on.

In this paper, we focus on the scenario in which the adversary acts in an environment where this interaction is modeled as a Markov decision process (MDP) [6]. The adversary’s aim is to collect rewards at each state of the MDP and the defender tries to minimize the accumulated reward through deception. Many existing approaches for deception rely on a rational adversary with sufficient memory and computation power to find its optimal policy [7, 1]. However, deceiving an adversary with only bounded rationality [8], i.e., one whose decisions may follow certain rules that deviate from the optimal action [9], has not been adequately studied so far. Deceiving an adversary with bounded rationality finds, for example, its application in intrusion detection and protection [10] or public safety [11]. Different from obfuscating sensitive system information to the adversary [12, 13], by deception, we mean that the defender optimally assigns a limited resource to each state, such that the expected cost from defender’s perspective (or equivalently, the reward for the adversary) incurred by an adversary can be minimized, even though the adversary is expecting more based on his cognitively biased view of rewards.

To deceive a human more effectively, it is essential to understand the human’s cognitive characteristics and what affects his decisions (particularly with stochastic outcomes). Works in behavior psychology, e.g. [14], suggested that humans’ decision-making follows intuition and bounded rationality. Empirical evidence has shown that humans tend to evaluate gains and losses differently in decision-making [15]. Humans tend to over-estimate the likelihood of low-probability events and underestimate the likelihood of high-probability events in a nonlinear fashion [16, 15]. Risk-sensitive measures, such as those in the so-called prospect theory [16], capture such biases and are widely used in psychology and economics to characterize human preferences. Furthermore, humans tend to make decisions that are often sub-optimal [17]. It is generally believed that such sub-optimality is the result of intuitive decisions or preferences that happen automatically and quickly without much reflection [17, 14]. Human decisions are subject to stochasticity due to the limited computational capacity and inherent noise [18]. Consequently, human decisions are often cognitively biased (have a different reward mechanism), probabilistic (have a stochastic action selection policy) and memoryless (only depends on the current state). These are the very characteristics of human decision-making we expect to account for in reward-based deception.

This paper investigates how one can deceive a human adversary by optimally allocating limited resources to minimize his rewards. We model the environment as an MDP to capture the choices available to a human decision-maker and their probabilistic outcomes. We consider opportunistic human adversaries, i.e., they usually do not have significant planning and only act based on immediately available rewards [11]. We describe the human adversary’s policy to select different actions following the prospect theory and bounded rationality [8]. We model both the adversary’s perceived reward and defender’s cost (equivalently, the adversary’s reward from the defender’s point of view) as functions of the resources available at each state of the MDP. Additionally, we define a subset of the states in the MDP as sensitive states that the human adversary should be kept from visiting.

We then formulate the optimal resource allocation problem as a signomial program (SP) to minimize the defender’s cost. SPs are a special form of nonlinear programming problems, and they are generally nonconvex. Solving nonconvex NLPs is NP-hard [19] in general, and a globally optimal solution of an SP cannot be computed efficiently. SPs generalize geometric programs (GP), which can be transformed into convex optimization problems and then can be solved efficiently [20]. In this paper, we approximate the proposed SP to a GP. In numerical experiments, we show that this approach obtain locally optimal solutions of the SP efficiently by solving a number of GPs. We demonstrate the approach with a problem on the assignment of police patrol hour against opportunistic criminals [11].

The problem we study is closely related to the Stackelberg security game (SSG) which consists of an attacker and a defender that interact with each other. In SSG, the defender acts first with limited resources and then the attackers play in response [21]. SSG is a popular formalism to study security problems against human adversaries. Early efforts focused on one-shot games where an adversary can only take one move [22] without considering human’s bounded rationality. Then repeated SSG was considered in wildlife security [10] and fisheries [23] where the defender and the adversary can have repeated interaction. However, neither of these papers considered how a human perceives probabilities, where the existence of nonlinear probability weighting curves is a well-known result in prospect theory [16]. Such phenomenon was taken into account in [24] and [25]. But [24] only studied one-shot games and [25] did not consider the adversaries may move from place to place.

The rest of this paper is organized as the following. We first provide the necessary preliminaries for stochastic environment modeling, human cognitive biases and decision-making in Section II. Then we formulate the human deception problem in terms of resource allocation in Section III and show that it can be transformed into a signomial program in Section IV. We propose the computational approach to solve the signomial program in Section V. Section VI shows simulations results and discusses their implications. We conclude our paper and discusses possible future directions in Section VII.

II Preliminaries

II-A Monomials, Posynomials, and Signomials.

Let $V=\{x_{1},\ldots,x_{n}\}$ be a finite set of strictly positive real-valued variables. A monomial over $V$ is an expression of the form

[TABLE]

where $c\in\mathbb{R}^{+}$ is a positive coefficient, and $a_{i}\in\mathbb{R}$ are exponents for $1\leq i\leq n$. A posynomial over $V$ is a sum of one or more monomials:

[TABLE]

If $c_{k}$ is allowed to be a negative real number for any $1\leq k\leq K$, then the expression (2) is a signomial.

This definition of monomials differs from the standard algebraic definition where exponents are positive integers with no restriction on the coefficient sign. A sum of monomials is then called a polynomial.

II-B Nonlinear programs.

A general nonlinear program (NLP) over a set of real-valued variables $V$ is

[TABLE]

where $f$, $g_{i}$, and $h_{j}$ are arbitrary functions over $V$, and $m$ and $p$ are the number of inequality and equality constraints of the program respectively.

II-C Signomial programs and geometric programs.

A special class of NLPs known as signomial programs (SP) is of the form (3)–(5) where $f$, $g_{i}$ and $h_{j}$ are signomials over $V$, see Def. II-A. A geometric program (GP) is an SP of the form (3)–(5) where $f,g_{i}$ are posynomial functions and $h_{j}$ are monomial functions over $V$. GPs can be transformed into convex programs [20, §2.5] and then can be solved efficiently using interior-point methods [26]. SPs are non-convex programs in general, and therefore there is no efficient algorithm to compute global optimal solutions for SPs . However, we can efficiently obtain local optimal solutions for SPs in our setting, as shown in the following sections.

In this paper, the adversary with bounded rationality moves in an environment modeled as a Markov decision process (MDP) [6].

II-D Markov Decision Processes.

A (MDP) is a tuple $\mathcal{M}=(S,\nu,A,T,U)$ where

• •

$S$ is a finite set of states;

• •

$\nu:S\rightarrow[0,1]$ is the initial state distribution;

• •

$A$ is a finite set of actions;

• •

$T(s,a,s^{\prime}):=P(s^{\prime}|s,a)$. That is, the probability of transiting from $s$ to $s^{\prime}$ with action $a$; and

• •

$U(s)\in\mathbb{R}^{+}$ is the utility function that assigns resources with a quantity $U(s)$ to state $s$.

At each state $s$, an adversary has a set of actions available to choose. Then the nondeterminism of the action selection has to be resolved by a policy $\pi$ executed by the adversary. A (memoryless) policy $\pi:S\times A\rightarrow[0,1]$ of an MDP $\mathcal{M}$ is a function that maps every state action pair $(s,a)$ where $s\in S$ and $a\in A$ with probability $\pi(s,a)$.

By definition, the policy $\pi$ specifies the probability for the next action $a$ to be taken at the current state $s$. A bounded rational adversary is often limited in memory and computation power, therefore we only consider the memoryless policies.

In an MDP, a finite state-action path is $\omega=s_{0}a_{0}s_{1}a_{1}...$, where $s_{i}\in S,a_{i}\in A$ and $T(s_{i},a_{i},s_{i+1})>0$. Given a policy $\pi$, it is possible to calculate the probability of such path $P_{\pi}(\omega)$ as

[TABLE]

III Reward-Based Deception

We assume that an adversary with bounded rationality moves around in an environment modeled as an MDP $\mathcal{M}=(S,\nu,A,T,U)$. When the adversary is at a state $s\in S$, from the defender’s point of view, the immediate reward for the human adversary (or equivalently, the cost for the defender) is

[TABLE]

which is a function of allocated resource $U(s)$. However, due to the bounded rationality and cognitive biases, the perceived immediate reward $R_{h}(s)$ at state $s$ by the adversary is a different function of $U(s)$, and is given by

[TABLE]

where $f$ is another function over $U$. For a given policy $\pi$, expected rewards $Q_{t}(s)$ at each state $s$ and time t with a finite time horizon $H$ can be evaluated as

[TABLE]

where $t=0...,H-1$, $Q_{H}(s)=R(s)$. Therefore, $Q_{t}$ represents the expect accumulated cost of the defender, or equivalently, expected rewards for the human adversary obtained from the policy $\pi$.

The defender’s objective is to optimally assign the resources to each state to minimize his cost (equivalently, the adversary’s reward) $Q$, where

[TABLE]

by designing the utility function $U$, where the resources are of limited quantity, i.e., $\sum_{s}U(s)=D$. Also imagine that there are set of sensitive states $S_{s}\subset S$ that the adversaries should be kept away from. Denote the set of paths that reach $S_{s}$ in $H$ steps as $\Omega$ such that for each $\omega\in\Omega$ where $\omega=s_{0}a_{0},...,s_{N}$, we require $N\leq H,s_{i}\notin S_{s},i<N$ and $s_{N}\in S_{s}$. In particular, given a policy $\pi$, $P(\diamondsuit^{\leq H}S_{s})$ can be calculated as

[TABLE]

Problem 1

Given an MDP $\mathcal{M}=(S,\nu,A,T,U)$, time horizon $H$, human reward function defined in (13) and the policy $\pi$, design reward function $U$ with a limited total budget $\sum_{s}U(s)=D$, such that $Q$ as defined in (8) can be minimized $\hat{s}$ and $P(\diamondsuit^{\leq H}S_{s})\leq\lambda$, which requires that the probability to reach $S_{s}$ in $H$ steps should be no larger than $\lambda\in[0,1]$, i.e.,

[TABLE]

Remark 1

Problem 1 studies how to optimally assign the reward to trick the adversary into thinking that his policy could obtain more rewards but in fact, the actual expected reward is minimized with a low probability of visiting sensitive states $S_{s}$.

III-A Human Adversaries with Cognitive Biases

To solve Problem 1, it is essential to find the adversary’s policy $\pi$. In this paper, we take human as the adversary with bounded rationality who is opportunistic, meaning that he does not have a specific attack goal nor plans strategically, but is flexible about his movement plan and seek opportunities for attacks [27]. Those attacks may incur rewards to the human adversary and consequently certain costs for the defender. The process of human decision-making typically follows several steps [28]. First, a human recognizes his current situation or state. Second, he will evaluate each available action based on the potential immediate reward it can bring. Third, he will select an action following some rules. Then he will receive a reward and observe a new state. In this section, we will introduce the modeling framework for the second and third step.

For a human with bounded rationality, the value of a reward from an action is a function of the possible outcomes and their associated probabilities. The prospect theory developed by Kahneman and Tversky [16] is a frequently used modeling framework to characterize the reward perceived by a human. Prospect theory claims that humans tend to over-estimate the low probabilities and underestimate the high probabilities in a nonlinear fashion. For example, between winning $100$ dollar with $\frac{1}{100}$ probability and nothing else, or $1$ dollar with probability $1$, humans tend to prefer the former, even though both have the same expectation.

Given $X$ as the discrete random variable that has a finite set of outcomes $O$, a general form of prospect theory utility $V(X)$ (i.e. the reward anticipated by a human) is the following.

[TABLE]

where $v(x)\in\mathbb{R}$ denotes the reward perceived by a human from the outcome $x$. The probability $p(x)$ to get the outcome $x$ is weighted by a nonlinear function $w$ that captures the human tendency to over-estimate low probabilities and under-estimate high probabilities.

The expected immediate reward $r_{a}(s)$ to perform an action $a$ at state $s$ is

[TABLE]

However, according to prospect theory, from a human’s perspective, the perceived expected immediate reward $r_{a}^{h}(s)$ is different. Let $X_{s,a}$ be the random variable for the outcome $O_{s,a}$ of executing action $a$ at state $s$. We have $O_{s,a}=\{x_{s}^{\prime}|T(s,a,s^{\prime})>0\}$ where $x_{s^{\prime}}$ denotes the event that the state transits from $s$ to $s^{\prime}$ with an action $a$. The distribution of $X_{s,a}$ is defined as follows.

[TABLE]

The human perceived reward $v(x_{s^{\prime}})$ for the outcome $x_{s^{\prime}}$ depends on $U(s^{\prime})$ received from reaching the state $s^{\prime}$, which is denoted by

[TABLE]

As a result, $r_{a}^{h}(s)$ is denoted by

[TABLE]

An empirical form of $w$ is the following [16].

[TABLE]

Given an MDP as depicted in Figure 1, where $S=\{s_{0},\ldots,s_{4}\}$, $A=\{a,b\}$. We assume that $R(s)=U(s)$, $R_{h}(s)=U(s)^{0.88}$, $\gamma=0.6$ in (14). It can be found from (12) and (13) that $r_{a}(s_{0})=2.3,r_{a}^{h}(s_{0})=2.0678,r_{b}(s_{0})=2.5$ and $r_{b}^{h}(s_{0})=1.8617$. Since $r_{a}^{h}(s_{0})>r_{b}^{h}(s_{0})$, suppose a human is at $s_{0}$, from human’s perspective, he will prefer the action $a$. However, $r_{a}(s_{0})<r_{b}(s_{0})$ which indicates that action $a$ actually has more expected immediate rewards.

Remark 2

In this example, the rewards are already given, and it can be seen that the human could make a sub-optimal decision. It illustrates how cognitive bias can deviate the human behavior from optimal.

After evaluating the outcome of each candidate action $a$ by $r_{a}^{h}(s)$, a human then needs to make an action selection. Humans are known to only have quite limited cognitive capabilities. Human’s policy $\pi$ to choose an action can be described as a random process that biases toward the actions of high $r_{a}^{h}(s)$, such that

[TABLE]

where $\pi(s,a)$ denotes the probability of executing the action $a$ at state $s$. Such a bounded rational behavior has been observed in humans, such as urban criminal activities [29]. Intuitively, it implies that human selects the action $a$ opportunistically at each state $s$ with the probability proportional to the perceived immediate reward $r^{h}_{a}(s)$.

Now we are ready to redefine Problem 1 as follows.

Problem 2

Solve Problem 1 for $\pi$ defined as (15).

IV Signomial Programming Formulation

Given an MDP $\mathcal{M}$, time horizon $H$, human reward function and policy as defined in (13) and (15), the solution of the Problem 1 can be computed by solving the following signomial program. The $g$ and $f$ are assumed to be monomial functions of $U$ for our solution method.

[TABLE]

where variables $R(s)$ are for rewards in each state $s$, $U(s)$ are for utilities in each state $s$, $\pi(s,a)$ are for the probability of taking action $a$ in state $s$ are for each state and action, $Q_{t}(s)$ are for the expected reward of the state $s$ and time step $t$, and $P_{t}(s)$ are for the probability of reaching the set of target states $S_{s}$ in each state $s$ and time step $t$.

The objective in (16) minimizes the accumulated expected reward from the initial state distribution $\nu(s)$ over a time horizon $H$. In (17), we compute $Q_{t}(s)$ by adding the immediate reward in state $s$ and the expected reward of the successor states according to the policy variables $\pi(s,a)$ for each action $a$. The probability of reaching each successor state $s^{\prime}$ depends on the policy variables $\pi(s,a)$ in each state $s$ and action $a$. Similar to the constraint in (17), the variables $P_{t}(s)$ are assigned to the probability of reaching the set of target states $S_{s}$ from state $s$ and time step $t$ in (18).

The probability of reaching any state $s\in S_{s}$ in each horizon from the states in $S_{s}$ is set to 1 as in (19). The constraint in (20) assures that the probability of reaching any state $s\in S_{s}$ from the initial state distribution $\nu(s)$ is less than $\lambda$. The constraint in (21) computes the policy using the model in (15). We give the relationship between rewards and utilities in (22). Finally, (23) gives the total budget for utilities.

The constraint in (17) and (18) are convex constraints, because the functions in the right hand sides are posynomial functions, and the functions in the left hand sides are monomial functions. The constraints in (19) and (20) are affine constraints, therefore they are convex. The constraints in (21) and (23) are equality constraints with posynomials, therefore they belong to the class of signomial constraints, and they are not convex. In the literature, there are various methods to deal with the nonconvex constraints to obtain a locally optimal solution including sequential convex programming, convex-concave programming, branch and bound or cutting plane methods [30, 20, 31].

V Computational Approach for the Signomial Program

In this section, we discuss how to compute a locally optimal solution efficiently for Problem 1 by solving the signomial program in (16)–(23). We propose a sequential convex programming method to compute a local optimum of the signomial program in (16)–(23), following [20, §9.1], solving a sequence of GPs. We obtain each GP by replacing signomial constraints in equality constraints of the SGP signomial program in (16)–(23) with monomial approximations of the functions.

V-A Monomial approximation

Given a posynomial $f$, a set of variables $\{x_{1},\dots,x_{n}\}$, and an initial point $\hat{x}$, a monomial approximation [20] $\hat{f}$ for $f$ around $\hat{x}$ is

[TABLE]

Intuitively, a monomial approximation of a posynomial $f$ around an initial point $\hat{x}$ corresponds to an affine approximation of the posynomial $f$. Such an approximation is provided by the first order Taylor approximation of $f$, see [20, §9.1] for more details.

For a given instantiation of the utility and policy variables $U(s)$ and $\pi(s,a)$, we approximate the SP in (16)–(23) to obtain a GP as follows. We first normalize the utility values to ensure that they sum up to $D$. Then, using those utility values, we compute the policy according to constraint in (21). After the policy comptutation, we compute a monomial approximation of each posynomial term in the constraints (21) and (23) around the previous instantiation of the utility and policy variables. After the approximation, we solve the approximate GP. We repeat this procedure until the procedure converges.

One key problem with this approach is, we require an initial feasible point to the signomial problem in (16)–(23), which may be hard to find because of the reachability constraint in (20). Therefore, we introduce a new variable $\tau$ and we replace the reachability constraint in (20) by the following constraints:

[TABLE]

By replacing the reachability constraint, we ensure that any initial utility function and policy is feasible to the signomial program in (16)–(25). To enforce the feasability of the reachability constraint in (20), we change the objective in (16) as follows:

[TABLE]

where $\delta$ is a positive penalty parameter that determines the violation rate for the soft constraint in (24). In our formulation, we increase $\delta$ after each iteration to satisfy the reachability constraint.

We stop the iterations when the change in the value of $Q$ is less than a small positive constant $\epsilon$. Intuitively, $\epsilon$ defines the required improvement on the objective value for each iteration; once there is not enough improvement, the process terminates.

VI Numerical Experiment

Let us consider an urban security problem, where a criminal plans his next move randomly based on his local information on the nearby locations that are protected by police patrols. Such a criminal is opportunistic, i.e, he is not highly strategic by conducting careful surveillance and rational planning before making moves. It is known that this kind of opportunistic adversaries contribute to the majority of the urban crimes [32]. For prevention and protection, each location should be assigned a certain police patrol hours. Due to the limited amount of police resources, the total number of patrol hours is limited as well.

Figure 2 shows $35$ intersections in San Francisco, CA with $7$ rows and $5$ columns. We use an MDP $\mathcal{M}=(S,\nu,A,T,U)$ to describe the network of the set of intersections $S$. The number $C(s)$ of crimes that occurred in the first four weeks of October, 2018 within 500 feet of each interaction $s$ is shown in Figure 2. The crime data are obtained from https://www.crimemapping.com/map/ca/sanfrancisco. The criminal can choose to move left, right, up or down to the immediate neighboring intersections. Consequently, there are four actions available. The execution of each action will lead the human to its intended neighborhood of the intersection with a high probability ($\geq 0.95$) and small probability to other neighboring intersections to account for unexpected change of movement plan.

Initially, the criminal has equal probability to appear at any state, i.e., $\nu(s)=\frac{1}{35}$ for any $s\in S$. The utility $U(s)$ denotes the number of police patrol hours that should be allocated to the vicinity of each intersection. The total number of police patrol hours is $D=\sum U(s)=400$. If a location $s$ is assigned with $U(s)$ patrol hours, its reward to the criminal (equivalently, the cost to the defender) is

[TABLE]

Intuitively, it means that the reward to the human adversary, from the defender’s point of view, is proportional to the crime rate indicated by $C(s)$ and inversely proportional to the police patrol hours. The reward from the human adversary’s view is evaluated as

[TABLE]

which is a function commonly seen in the literature to describe how human biases the reward [15].

Initially, the criminal is at $s$ with probability $\nu(s)$, where he tries to plan his move over the next $H$ steps. The objective is to assign the police patrol hours to each state, such that the expected accumulated reward in $H$ steps received by the criminal is minimized. The sensitive states $S_{s}=\{3,14,33\}$ should be visited with a probability no larger than $\lambda=0.3$, i.e.

[TABLE]

The sensitive states are also shown as blue circles in Figure 2.

We formulate the problem as a signomial program. From an initial uniform utility distribution, we instantiate the policies and reward functions. Then, from the initial values, we linearize the signomial program in (16)–(26) to a geometric program. We parse the geometric programs using the tool GPkit [33], and solve them using the solver MOSEK [34]. We set $\epsilon=10^{-4}$ for convergence tolerance. All experiments were run on a 2.3 GHz machine with 16 GB RAM. The procedure converged after 32 iterations for a problem with horizon length $T=20$ in 230.06 seconds. The expected reward $Q$ from the initial state distribution is 117.15, and the reachability probability of the sensitive states from the initial state distribution is $0.192$, which satisfies the reachability specification.

The result is shown in Figure 3. Different colors at each intersection show the number of patrol hours, i.e, the resource $U(s)$, assigned to each location s. In Figure 3, $U(s)$ is shown with a logarithmic scale for better illustration. As the color bar at the bottom of the figure indicates, the closer the color at each location is to the right side of this bar, the higher patrol hours are assigned. For example, the state at $(3,1)$ (the third state from the first row), where $C(s)=48$ gets assigned patrol hour equals $106$ which is approximately $2$ in logarithmic scale. Therefore, its color is yellow in Figure 3 as indicated by the right tip of the color bar. Together with Figure 2, it can be observed that sensitive places and places with a higher number of crimes get assigned more patrol hours. Consequently, the rewards at those states are fairly low to discourage the criminal from visiting it. The cost at each location is proportional to the crime rate and inversely proportional to the police patrol hours. The patrol hours assigned to each place intends to minimize the expected cost incurred by the human adversary.

VII Conclusion

This paper introduces a general framework for deceiving adversaries with bounded rationality in terms of the obtained reward minimization. Leveraging the cognitive bias of the human from well-known prospect theory, we formulate the reward-based deception as a resource allocation problem in Markov decision process environment and solve as a signomial program to minimize the adversary’s expected reward. We use police patrol hour assignment as the illustrative example and show the validity of our propose solution approach. It opens doors for further research on the topic to consider the scenarios where defender can move around and react to the human adversaries in real time, and the human adversary has a learning capability to adapt the defender’s deceiving policy.