Imitative Follower Deception in Stackelberg Games

TL;DR

This paper investigates how a leader in Stackelberg games can mitigate losses caused by followers who strategically imitate other types, proposing models and algorithms to design robust leader strategies against such deception.

Contribution

The paper introduces a model of follower deception in Stackelberg games and analyzes the computational complexity of deriving optimal leader policies under deception.

Findings

Leader can reduce losses with carefully designed policies.

Computational complexity varies across different model variants.

Follower deception significantly increases the difficulty of strategy computation.

Abstract

Information uncertainty is one of the major challenges facing applications of game theory. In the context of Stackelberg games, various approaches have been proposed to deal with the leader's incomplete knowledge about the follower's payoffs, typically by gathering information from the leader's interaction with the follower. Unfortunately, these approaches rely crucially on the assumption that the follower will not strategically exploit this information asymmetry, i.e., the follower behaves truthfully during the interaction according to their actual payoffs. As we show in this paper, the follower may have strong incentives to deceitfully imitate the behavior of a different follower type and, in doing this, benefit significantly from inducing the leader into choosing a highly suboptimal strategy. This raises a fundamental question: how to design a leader strategy in the presence of a deceitful follower? To answer this question, we put forward a basic model of Stackelberg games with (imitative) follower deception and show that the leader is indeed able to reduce the loss due to follower deception with carefully designed policies. We then provide a systematic study of the problem of computing the optimal leader policy and draw a relatively complete picture of the complexity landscape; essentially matching positive and negative complexity results are provided for natural variants of the model. Our intractability results are in sharp contrast to the situation with no deception, where the leader's optimal strategy can be computed in polynomial time, and thus illustrate the intrinsic difficulty of handling follower deception. Through simulations we also examine the benefit of considering follower deception in randomly generated games.

Tables

Table 1. Table 1: Leader utility yielded by robust solutions as a ratio to the utility under the optimistic tie-breaking assumption, with ϵ italic-ϵ \epsilon varying from 10 − 5 superscript 10 5 10^{-5} to 10 − 1 superscript 10 1 10^{-1} . Results of OptX (IC) - large are obtained on large instances with α = 0.5 𝛼 0.5 \alpha=0.5 , | Θ | = 50 Θ 50 |\Theta|=50 , n = m = 20 𝑛 𝑚 20 n=m=20 ; all other results are obtained on instances with α = 0.5 𝛼 0.5 \alpha=0.5 , | Θ | = 5 Θ 5 |\Theta|=5 , n = 5 𝑛 5 n=5 and m = 10 𝑚 10 m=10 . Each entry is the average of 100 runs. ( ∗ * ) In 9 out of 100 runs of this setting there is no feasible solution.

	${10}^{-5}$	${10}^{-4}$	${10}^{-3}$	${10}^{-2}$	${10}^{-1}$
Opt	1.00	1.00	1.00	0.97	0.63
Opt (IC)	1.00	1.00	1.00	0.97	0.61∗
OptX	1.00	1.00	1.00	0.97	0.66
OptX (IC)	1.00	1.00	1.00	0.98	0.67
OptX (IC) - large	1.00	1.00	1.00	0.98	0.79

Equations

$$⟨{\mathbf{x}}^{\ast },j^{\ast }⟩\in \arg {\max }_{\mathbf{x}\in {\Delta }_m,j\in \text{BR}(\mathbf{x})}{u}^{\text{L}}(\mathbf{x},j).$$

$$U^{\text{L}}(\sigma ):=\sum _{\theta \in \Theta }{\pi }_{\theta }\cdot u^{\text{L}}({\mathbf{x}}^{\hat{\theta }(\sigma )},j^{\hat{\theta }(\sigma )}).$$

$$u^{\text{L}}({\mathbf{x}}^{\beta },j^{\beta })$$

$$u_{\theta }^{\text{F}}({\mathbf{x}}^{\theta },j^{\theta })\ge u_{\theta }^{\text{F}}({\mathbf{x}}^{\theta },j)$$

$$u_{\beta }^{\text{F}}({\mathbf{x}}^{\beta },j^{\beta })\ge u_{\beta }^{\text{F}}({\mathbf{x}}^{\beta },j)$$

$$u_{\theta }^{\text{F}}({\mathbf{x}}^{\beta },j^{\beta })\ge u_{\theta }^{\text{F}}({\mathbf{x}}^{\theta },j^{\theta })$$

$${\mathbf{x}}^{\theta },{\mathbf{x}}^{\beta }\in {\Delta }_m$$

$$U^{\text{L}}({\sigma }_{{\theta }^{\ast }})\ge {\pi }_{{\theta }^{\ast }}\cdot {\mu }_{{\theta }^{\ast }}^{\text{L}}\ge \frac{1}{|\Theta |}\sum _{\theta }{\pi }_{\theta }\cdot {\mu }_{\theta }^{\text{L}}\ge \frac{1}{|\Theta |}\sum _{\theta }{\pi }_{\theta }\cdot u^{\text{L}}({\mathbf{x}}^{\hat{\theta }(\varsigma )},j^{\hat{\theta }(\varsigma )})=\frac{1}{|\Theta |}\cdot U^{\text{L}}(\varsigma ).$$

$$\textstyle\sum_{\theta}\pi_{\theta}\ \cdot{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mu_{\theta}^{\textit{{L}}}}$$

$$\textstyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mu_{\theta}^{\textit{{L}}}}\leq\sum_{j}p_{j}^{\beta}\cdot u^{\textit{{L}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{x}^{\beta}},j)+(1-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}y_{\beta}^{\theta}})\cdot M$$

$$\textstyle p_{j}^{\theta}\cdot\left[u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{x}^{\theta}},j)-u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{x}^{\theta}},k)\right]\geq 0$$

$$\textstyle\sum_{j}p_{j}^{\beta}\cdot u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{x}^{\beta}},j)-\sum_{j}p_{j}^{\gamma}\cdot u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{x}^{\gamma}},j)\geq-(1-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}y_{\beta}^{\theta}})\cdot M$$

$$\textstyle\sum_{i}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}x_{i}^{\theta}}=\sum_{j}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{j}^{\theta}}=\sum_{\beta}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}y_{\beta}^{\theta}}=1$$

$$\textstyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{x}^{\theta}}\in[0,1]^{m},\ \ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}y_{\beta}^{\theta}}\in\{0,1\},\ \ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{j}^{\theta}}\in\{0,1\}$$

$$\sum _{⟨\mathbf{x},j⟩\in \mathcal{O}}{p}_{⟨\mathbf{x},j⟩}\cdot u^{\text{L}}(\mathbf{x},j)$$

$$\textstyle\sum_{\theta}\pi_{\theta}\ \sum_{j}u^{\textit{{L}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{\mathbf{x}}_{j}^{\theta}},j)$$

$$\textstyle\sum_{j}u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{\mathbf{x}}_{j}^{\theta}},j)-\sum_{j}u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{\mathbf{x}}_{j}^{\beta}},j)\geq 0$$

$$\textstyle u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{\mathbf{x}}_{j}^{\theta}},j)-u_{\theta}^{\textit{{F}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{\mathbf{x}}_{j}^{\theta}},k)\geq 0$$

$$\textstyle\sum_{i}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{x}_{ji}^{\theta}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{j}^{\theta}}$$

$$\textstyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{\mathbf{x}}_{j}^{\theta}}\in\mathbb{R}_{\geq 0}^{m},\ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}(p_{1}^{\theta},\dots,p_{n}^{\theta})}\in\Delta_{n}$$

Full text

1 Introduction

Recently, there is a growth of interest in Stackelberg games in the AI community. This trend is driven in part by a number of high-impact real-world applications in security domains Tambe (2011). Beyond that, Stackelberg game models also find many noteworthy applications in other problems, such as principal-agent contract design Page (1991); Van Long and Sorger (2010) and exam design for large-scale tests Li and Conitzer (2017). In a Stackelberg game, a leader commits to a mixed strategy and a follower best-responds after observing the leader’s strategy. The Stackelberg equilibrium yields the optimal strategy the leader can commit to in this framework. As in many other game-theoretic applications, a key real-world challenge facing applications of Stackelberg games is that the leader may not have full information about the follower’s payoffs for computing the equilibrium. To address this issue, various approaches have been proposed: When the leader can estimate the follower’s payoffs to within certain intervals, leader strategies that are robust against small interval uncertainties are computed Letchford et al. (2009); Kiekintveld et al. (2013); Nguyen et al. (2014); When the leader knows a distribution about the follower’s payoffs/types, Bayesian Stackelberg equilibria are studied to maximize the leader’s average utility Conitzer and Sandholm (2006); Paruchuri et al. (2008); Jain et al. (2008); When the leader can interact with the follower, algorithms are designed to learn the optimal leader strategy to commit to from the follower’s best responses in the interaction Letchford et al. (2009); Blum et al. (2014); Haghtalab et al. (2016); Roth et al. (2016); Peng et al. (2019).

Despite their differences, the above approaches share a common and crucial step — obtaining payoff-relevant information about the follower. However, knowing the leader’s attempt at information gathering, a strategic follower would be incentivized to intentionally distort information learned by the leader, in particular by deceitfully imitating the behavior of a different follower type — a phenomenon we term imitative follower deception. Unfortunately, all aforementioned approaches ignore the possibility of a deceitful follower and adopt a simplistic assumption that any follower information the algorithm gathers is truthful.

As we will show in this paper, algorithms designed under the above assumption can be easily manipulated by the follower and may produce highly suboptimal leader strategies as a result. Consider perhaps the most basic learning setting where the follower has an uncertain type (i.e., a full set of payoff parameters) $\theta\in\Theta$. Knowing the set $\Theta$ of all possible types but not the actual $\theta$, the leader wants to learn the optimal strategy against this follower. If the follower is truthful, the optimal strategy the leader can learn would be the strong Stackelberg equilibrium. However, if the follower behaves completely according to the payoff parameters of another type $\theta^{\prime}$ (he may indeed have such an incentive, as we will show), the leader can only learn the optimal strategy against the fake type $\theta^{\prime}$, which may be highly suboptimal. A fundamental question then is: what is the optimal strategy the leader can learn or design when facing a deceitful follower?

In this paper, we put forward a basic Stackelberg model in an attempt to formalize this question and aim to understand how imitative follower deception affects the leader’s choice of strategies. We note that, even though there might be other forms of follower deception as well, imitative deception is arguably the simplest and the most natural deception approach, which we expect to happen the most often in practice. Our model is Bayesian by nature: the type of the follower is drawn from a finite set $\Theta$; the follower knows his true type, but the leader only has a probabilistic belief about it. To play against a deceitful follower, the leader commits to a policy — a “menu” that specifies a mixed strategy to play for each (learned) follower type. Knowing the leader’s commitment, a follower of true type $\theta\in\Theta$ can imitate another type $\theta^{\prime}\in\Theta$, behaving consistently according to the payoffs of $\theta^{\prime}$ to mislead the leader into learning the fake type $\theta^{\prime}$. The leader then plays the strategy specified by the committed policy for $\theta^{\prime}$. The optimal leader policy maximizes the leader’s expected utility, taking into account the follower’s optimal imitating strategy in the loop; this is precisely the optimal leader strategy that can be learned in the deceptive setting.

1.1 Our Contribution

We make the following contribution. In Section 2, we provide a motivating example to illustrate the mechanisms of imitative follower deception, the loss it brings to the leader, and how the leader can reduce the loss with carefully designed strategies. Then, we formalize the model in Section 3.

In Section 4, we study the hardness of handling follower deception and draw a relatively complete picture of the problem’s complexity landscape. We consider settings both with and without incentive compatibility (IC) constraints; the former requires the leader strategy to incentivize the follower to report his true type while the latter does not. On the negative side, we show that it is hard even to approximate the optimal leader policy within any meaningful ratio unless P = NP, with or without IC constraints; the ratios are also shown to be tight. The results indicate that deception is the fundamental reason of the intractability as, when there is no deception, the problem can be solved in polynomial time simply by computing the optimal commitment for each follower type separately. Despite the computational barriers, we provide an MILP (mixed integer linear program) formulation as a practical approach to compute the optimal leader policy, from which a fixed-parameter tractability with respect to the number of follower types can be derived.

In Section 5, we further observe that the leader can improve her utility using a mixed policy, that samples from a distribution a strategy to play for each follower type. We study the respective computational problems above with mixed policies and find that, in addition to the utility improvement they bring, the optimal mixed policy can be computed in polynomial time when IC constraints are imposed; though, the problem remains hard to approximate without IC.

In Section 6, through simulations, we compare the improvement of leader utility by our approaches and other benchmark approaches in randomly generated games, and examine the benefit of considering follower deception. We conclude in Section 7 with a short discussion on future directions of this work.

1.2 Additional Applications and Related Work

Besides furthering our understanding about the best leader strategy that can be learned when facing a deceitful follower, our model can also be applied directly to a number of applications. For example, in principal-agent contract design (a widely studied Stackelberg model in economics Page (1991); Van Long and Sorger (2010); con (2017)), a principal may present a menu of contracts to an uncertain agent and ask the agent to choose one that matches his true type. Naturally, the principal’s menu must take into account the agent’s misreport of his type. Similar ideas apply to exam design for large-scale tests, e.g., for MOOCs (massive open online courses), which have also been modeled as Stackelberg games Li and Conitzer (2017). In these situations, the tester may need to use different exams for different types of exam takers who usually come from various education backgrounds and have different learning objectives. Again, the exam design will also need to take into account the exam taker’s possible misreport about their type to strategically conceal their strength in the hope of a preferred test.

Deception has been extensively studied in Stackelberg games, particularly in its applications in security domains Brown et al. (2005); Powell (2007); Zhuang and Bier (2010); Yin et al. (2013); Xu et al. (2015); Guo et al. (2017); Schlenker et al. (2018). However, all previous works focus on designing deceptive strategies for the leader (usually, a defender). In these models, it is the leader who has the informational advantage, whereas in our model the follower has the greater decision-pertinent knowledge and discloses it strategically. To the best of our knowledge, there has been very limited work studying follower deception in Stackelberg games. The most relevant to ours is perhaps Xu et al. (2016), which studies signaling in Bayesian Stackelberg games and also considers the possibility that a follower may strategically misreport his type. However, the question they study is completely different from ours — they design signaling schemes for different follower types whereas we design the leader’s mixed strategy for each follower type. The contrasting complexity between their model (polynomial-time solvable for normal form games) and ours (NP-hard) also highlights the intrinsic difference. Even more importantly, though, in our model type reporting takes the conceptual form of a behaviour inducing message. The common information asymmetry (see e.g. Rabinovich et al. (2015); Xu et al. (2015); Dughmi et al. (2016)) is reversed between the leader and the follower.

2 A Motivating Example

We illustrate how the follower’s deceptive behavior may result in highly suboptimal leader strategy, and how we can overcome this issue using carefully-designed leader strategies. For illustrative purpose, we use a real-world security game example, though the underlying phenomenon also generalizes to other applications.

A security game is a Stackelberg game played between a defender (the leader) and an attacker (the follower). Concretely, we consider a security agency who wants to protect two conservation areas, i.e., areas $1$ and $2$, from a poacher’s attack. The defender can only patrol one of these two areas, while the poacher chooses one area to attack. At different periods of the year, the poacher’s payoffs change due to price fluctuation of wildlife products on black markets. We capture this uncertainty using two possible types of poacher payoffs, i.e., type $A$ and $B$. Assume that the defender’s payoff is independent of the poacher’s type. The poacher knows his true type, but the defender only has a prior belief that each type shows up with probability $0.5$. (This can be obtained via, e.g., surveying past prices on black markets.) The payoff matrices of the defender (row player) and the two poacher types (column player) are shown below.

The only difference between these poacher types is their value for successful attacks, which is affected by wildlife product prices at that time period. Standard calculation shows that if the poacher has type $A$, the optimal defender strategy is $(3/4,1/4)$, i.e., patrolling areas $1$ and $2$ with probability $3/4$ and $1/4$, respectively, resulting in poacher utility [math] at both areas. By the standard assumption, the poacher thus breaks the tie in favor of the defender and attacks area $1$,111This assumption is without loss of generality. The corresponding solution concept, the strong Stackelberg equilibrium, remains the most widely adopted solution concept in the literature of Stackelberg games (see e.g., Osborne and Rubinstein (1994); Conitzer and Sandholm (2006); Tambe (2011)). Normally, via infinitesimal strategy variations, the leader can induce the follower to play any best response action to the leader’s benefit. For example, the defender can play $(\frac{3}{4}-\epsilon,\frac{1}{4}+\epsilon)$ with $\epsilon\to 0$, so that the poacher will strictly prefer to attack area 1 while the change to the defender’s utility can be arbitrarily small. Our empirical evaluation in Section 6 will further confirm the robustness of our solutions against the tie-breaking issue. yielding defender utility $1/2$. Similarly, the optimal defender strategy is $(1/2,1/2)$ against a type-$B$ poacher, which induces the poacher to also attack area $1$ and yields defender utility [math].

Ideally, the defender would like to play the optimal patrolling strategy against the poacher’s true type at any time which, however, is unknown to the defender. Nevertheless, if the poacher were to behave truthfully according to his type, the defender can easily learn the poacher’s type, e.g., by observing their (different) best responses to strategy $(3/5,2/5)$. Indeed, previous approaches for learning the optimal leader strategy rely crucially on the assumption that the follower will truthfully respond (e.g., Letchford et al. (2009); Blum et al. (2014); Haghtalab et al. (2016); Roth et al. (2016)). However, in this example, a type-$A$ poacher has strong incentives to be untruthful: by imitating a type-$B$ poacher and reacting exactly according to the type-$B$ payoff structure, a type-$A$ poacher is able to induce the defender to play $(1/2,1/2)$ — the optimal strategy against a type-$B$ poacher — which results in true utility ${\frac{1}{2}\cdot(-1)}+{\frac{1}{2}\cdot(3)}=1$ for the type-$A$ poacher. This strictly improves upon his previous utility [math] of reacting truthfully. Simple calculation shows that a type-$B$ poacher does not benefit from deception and will behave truthfully. As a result, the defender will not be able to distinguish type $A$ from $B$ under the poacher’s strategic deception.

The issue raised above is due to the fact that the defender focuses solely on optimizing her utility without considering the poacher’s strategic behavior. We now show how the defender can overcome this issue by taking into account the poacher’s deception. It happens that in this example the optimal policy for the defender is to still play $(3/4,1/4)$ for type $A$ and $(1/2,1/2)$ for type $B$, but induce a type-$B$ poacher to break tie “against” the defender by attacking area $2$.222This tie breaking can be induced explicitly by playing $(\frac{1}{2}+\epsilon,\frac{1}{2}-\epsilon)$ for an infinitesimal $\epsilon$. This slightly decreases the defender’s utility against a type-$B$ poacher (from [math] to $\frac{1}{2}\cdot(-1)+\frac{1}{2}\cdot 0.99=-0.005$), but it erases the incentive of a type-$A$ poacher to imitate type $B$: if the type-$A$ poacher imitates $B$ to attack area $2$ under defender strategy $(1/2,1/2)$, his expected utility would become $-1$, which is worse than his utility [math] of behaving truthfully.

Remarks

(i) This example illustrates an intriguing phenomenon: when there is follower deception, it may be undesirable to always induce tie breaking in favor of the leader, since other tie breaking strategies may enforce more desirable follower behaviors. If the defender adopts this nuanced strategy, the poacher will have no incentive to deceive. Thus, from the perspective of machine learning, what we are designing can be viewed as the best strategy the leader can learn in the presence of follower deception.

(ii) One might wonder why the defender does not ignore the poacher’s deception by simply playing a single strategy against all types of poachers, which is precisely the optimal commitment in a Bayesian Stackelberg game. This reason is that our more sophisticated approach can achieve better defender utility. In this example, it can be computed that the defender would play $(1/2,1/2)$ in the Bayesian Stackelberg equilibrium, obtaining utility [math]. However, the strategy we designed above yields defender utility $\frac{1}{4}-\frac{1}{400}$. A more formal result regarding the utility improvement will be presented in Proposition 1.

(iii) The price of ignoring follower deception can be huge. Figure 1 illustrates an example in which the price — the ratio between the leader’s utilities when deception is ignored and when deception is properly dealt with — is unbounded. (Payoffs are normalized to be in $[0,1]$ for the ratio to be meaningful.) Nevertheless, perhaps counter-intuitively, follower deception is not always bad for the leader. One example is a Stackelberg game version of the prisoner’s dilemma illustrated in Figure 2.

3 The Model

3.1 Stackelberg Game Basics

A Stackelberg game is played between a leader and a follower. A normal-form Stackelberg game is given by two matrices $u^{\textit{{L}}},u^{\textit{{F}}}\in\mathbb{R}^{m\times n}$, which are the payoff matrices of the leader (row player) and follower (column player), respectively. We use $u^{\textit{{L}}}(i,j)$ to denote a generic entry of $u^{\textit{{L}}}$, and use $[m]=\{1,...,m\}$ to denote the set of the leader’s pure strategies (also called actions). A mixed strategy of the leader is a probabilistic distribution over $[m]$, denoted by a vector $\mathbf{x}\in\Delta_{m}=\{\mathbf{p}\geq 0:\sum_{i\in[m]}p_{i}=1\}$. With slight abuse of notation, we denote by $u^{\textit{{L}}}(\mathbf{x},j)$ the expected utility $\sum_{i\in[m]}x_{i}\cdot u^{\textit{{L}}}(i,j)$. Notation for the follower is the same by changing the labels. We will sometimes write a pure strategy $i$ in positions where a mixed strategy is expected, in which case it represents the same strategy in its mixed strategy form, i.e., a basis vector $\mathbf{e}_{i}$.

In a Stackelberg game, the leader moves first by committing to a mixed strategy $\mathbf{x}$, and the follower then best responds to $\mathbf{x}$. Without loss of generality, it is assumed that the follower’s best response is a pure strategy with ties broken in favor of the leader if there are multiple best responses.333See footnote 1. Under this assumption, the leader mixed strategy that maximizes her expected utility leads to a strong Stackelberg equilibrium (SSE), which is the standard solution concept of Stackelberg games. Formally, denote by $\textit{{BR}}(\mathbf{x}):=\arg\max_{j\in[n]}u^{\textit{{F}}}(\mathbf{x},j)$ the set of follower best responses to a leader strategy $\mathbf{x}$; a pair of strategies $\mathbf{x}^{*}$ and $j^{*}$ forms an SSE if

[TABLE]

3.2 Stackelberg Games with Imitative Follower Deception

We now describe a basic model that captures (imitative) follower deception; the reader may refer back to our motivating example as an instantiation of this model. We consider a leader who faces a follower with an uncertain type, which falls in a discrete set $\Theta$. Each type $\theta\in\Theta$ corresponds to a different follower payoff matrix $u_{\theta}^{\textit{{F}}}\in\mathbb{R}^{m\times n}$. For ease of presentation, we will assume that the leader’s payoff does not depend on $\theta$, though we remark that all our results easily generalize to the case with type-dependent leader payoffs. To model the leader’s prior knowledge regarding the follower’s type, we adopt the classic Bayesian perspective and assume that the leader has a prior distribution $\pi$, i.e., each type $\theta$ appears with probability $\pi_{\theta}$. Note that $\pi,\Theta$ and the payoff matrices are common knowledge.

We assume that the leader can commit to a policy — a “menu” that specifies a mixed strategy $\mathbf{x}^{\theta}$ to be played against each follower type $\theta$. In addition, we assume that a follower best response $j^{\theta}\in\textit{{BR}}_{{\theta}}(\mathbf{x}^{\theta})$ is also specified for each $\theta$ in the policy, where $\textit{{BR}}_{{\theta}}(\mathbf{x}):=\arg\max_{j}u_{{\theta}}^{\textit{{F}}}(\mathbf{x},j)$ now denotes the best response set of a type-$\theta$ follower against leader strategy $\mathbf{x}$; hence, in the case of a tie, the follower will be induced to play this specific response. Note that, same as in the motivating example, $j^{\theta}$ is not necessarily the one that maximizes the utility the leader obtains on follower type $\theta$; rather, the leader would prefer to induce the follower to a carefully chosen follower action that discourages follower deception to some extent and benefits the leader’s overall utility.

After observing the leader’s policy, a follower of type $\theta$ reports a type $\hat{\theta}$ to the leader, and the leader then plays the mixed strategy $\mathbf{x}^{\hat{\theta}}$ as specified by her policy. The follower then “best” responds to $\mathbf{x}^{\hat{\theta}}$ with $j^{\hat{\theta}}$, as if his type is $\hat{\theta}$. Naturally, the follower will report a type that maximizes his actual expected utility, i.e., $\hat{\theta}=\arg\max_{\beta\in\Theta}u_{\theta}^{\textit{{F}}}(\mathbf{x}^{\beta},j^{\beta})$, resulting in the leader to obtain utility $u^{\textit{{L}}}(\mathbf{x}^{\hat{\theta}},j^{\hat{\theta}})$. The reporting step in our model is straightforward in various applications (see Section 1), whereas in some other applications, reporting abstracts the process that the leader learns the follower’s type by interacting with the follower (as in the motivating example).

In summary, the game proceeds as follows:

The leader commits to a policy $\sigma=\{o^{\theta}\}_{\theta\in\Theta}$ that prescribes, for each reported follower type $\theta\in\Theta$, an outcome $o^{\theta}=\langle\mathbf{x}^{\theta},j^{\theta}\rangle$ such that $j^{\theta}\in\textit{{BR}}_{{\theta}}(\mathbf{x}):=\arg\max_{j}u_{{\theta}}^{\textit{{F}}}(\mathbf{x},j)$.

• 2.

After observing the leader policy $\sigma$, a follower of type $\theta$, who appears with probability $\pi_{\theta}$, reports a best type $\hat{\theta}(\sigma)=\arg\max_{\beta\in\Theta}u_{\theta}^{\textit{{F}}}(\mathbf{x}^{\beta},j^{\beta})$ that maximizes his actual utility; same as the standard SSE assumption, we assume the follower breaks ties by reporting a type in favor of the leader. This results in overall leader utility

[TABLE]

The computational task we examine is $\max_{\sigma}U^{\textit{{L}}}(\sigma)$. For convenience, we will also write $u_{\theta}^{\textit{{F}}}(o)=u_{\theta}^{\textit{{F}}}(\mathbf{x},j)$ and $u^{\textit{{L}}}(o)=u^{\textit{{L}}}(\mathbf{x},j)$ for an outcome $o=\langle\mathbf{x},j\rangle$.

Remarks

Our model can be viewed as a generalization of the classic Bayesian Stackelberg game (BSG) model where the leader’s policy is restricted to prescribe the same mixed strategy for all follower types Conitzer and Sandholm (2006). Yet, our model cannot be reduced to a classic BSG, a notable difference being that the leader’s strategy (policy) space in our model is non-convex due to the extra component $j^{\theta}$ in the prescribed outcomes whereas it is convex for a BSG.

3.3 Incentive Compatibility (IC)

One can impose in addition the IC constraints on $\sigma$ so that reporting truthfully is a (weakly) dominant strategy for every follower type, i.e., $u_{\theta}^{\textit{{F}}}(\mathbf{x}^{\theta},j^{\theta})\geq u_{\theta}^{\textit{{F}}}(\mathbf{x}^{\beta},j^{\beta})$ for any $\beta\in\Theta$. We will consider model variants both with and without IC constraints. The following result shows that an IC policy always exists, and even when IC constraints are imposed, our approach always achieves at least the leader utility in a Bayesian Stackelberg equilibrium. This echoes our observation in the motivating example.

Proposition 1**.**

There always exists an IC policy, and the optimal IC policy achieves at least the leader utility in a Bayesian Stackelberg equilibrium.

Proof.

Let $\mathbf{x}$ be the leader strategy in a Bayesian Stackelberg equilibrium (BSE), and let $j^{\theta}$ be the best response of a type-$\theta$ follower in the BSE for each $\theta$, i.e., $j^{\theta}\in\arg\max_{j\in\textit{{BR}}_{\theta}(\mathbf{x})}u^{\textit{{L}}}(\mathbf{x},j)$. A policy that prescribes $o^{\theta}=\langle\mathbf{x},j^{\theta}\rangle$ for each follower type $\theta\in\Theta$ is trivially IC because by reporting truthfully the follower will also be induced to play his true best response $j^{\theta}$ to $\mathbf{x}$. In addition, it offers the leader utility $\sum_{\theta\in\Theta}\pi_{\theta}\cdot u^{\textit{{L}}}(\mathbf{x},j^{\theta})$ — exactly what she obtains in the BSE. ∎

4 Computing the Optimal Policy

In this section, we study the complexity and algorithms for computing the optimal leader policy. We will refer to the problem of computing the optimal policy Opt-Policy and, when IC constraints are imposed, Opt-Policy-IC.

4.1 Hardness of Approximation

We show in Theorems 2 and 3 that it is NP-hard even just to approximate the optimal policy to within a meaningful ratio, with or without IC constraints, and even when the follower has only a small number of actions. The bounds in the inapproximability results are essentially tight by Theorem 4. In contrast, there is an efficient algorithm for a small follower type set, which we show in Theorem 5 after we present an MILP (mixed integer linear program)-based algorithm for the general case. Our approximation ratios are all multiplicative; by convention, we normalize the leader’s payoff parameters to be in $[0,1]$ in order for the ratios to be meaningful.

Theorem 2**.**

For any constant $\epsilon>0$, Opt-Policy does not admit any polynomial-time $\frac{1}{{(|\Theta|-1)}^{1-\epsilon}}$-approximation algorithm unless P = NP, even when the follower has only three actions.

Proof.

We show a reduction from the Max-Independent-Set problem, which asks for the size of the maximum independent set in a graph $G=(V,E)$. A set of nodes $V^{\prime}\subseteq V$ is an independent set of $G$ if no edge in $E$ connects any pair of nodes in $V^{\prime}$; an independent set is maximum if there exists no other independent set with a larger size. It is known that no polynomial-time $\frac{1}{{|V|}^{1-\epsilon}}$-approximation algorithm exists for Max-Independent-Set, unless P = NP Zuckerman (2006).

We construct a Stackelberg game with $|V|+1$ follower types $\Theta=\{\theta_{*}\}\cup\{\theta_{v}:v\in V\}$, where each $\theta_{v}$ corresponds to node $v$. Let $\pi_{\theta_{*}}=0$ and $\pi_{\theta}=\frac{1}{|\Theta|-1}$ for all other $\theta$. The leader has $2|V|+1$ actions $\{a_{v}\!:v\!\in\!V\}\cup\{b_{v}\!:v\!\in\!V\}\cup\{a_{0}\}$; and the follower has three actions $\{\mathsf{1^{F}},\mathsf{2^{F}},\mathsf{3^{F}}\}$. The payoffs are given in Figure 3, where $\mathcal{N}(v)$ denotes the set of neighbouring nodes of $v$, and all the empty entries are [math].

The following are a few observations about the game:

• •

For follower type $\theta_{*}$, $j=\mathsf{1^{F}}$ strictly dominates all the other actions. A follower can only be induced to play $j=\mathsf{1^{F}}$ when $\theta_{*}$ is reported, in which case the leader gets utility $1$.

• •

For each $\theta_{v}$, when they are reported, $j=\mathsf{1^{F}}$ cannot be induced by any leader strategy because it is strictly dominated by $j=\mathsf{2^{F}}$. As a result, the leader gets [math] once $\theta_{v}$ is reported.

• •

Given the above, the leader’s overall utility is proportional to the number of follower types $\theta_{v}$ that are motivated to report $\theta_{*}$.

Now for any independent set $V^{\prime}$ of size $k$ in $G$, we can construct the following leader policy $\sigma$: $o^{\theta_{*}}=\langle a_{0},\mathsf{1^{F}}\rangle$, $o^{\theta_{v}}=\langle a_{v},\mathsf{2^{F}}\rangle$ if $v\in V^{\prime}$, and $o^{\theta_{v}}=\langle b_{v},\mathsf{2^{F}}\rangle$ if $v\notin V^{\prime}$. Given this policy, all types $\theta_{v}$, $v\in V^{\prime}$, will find reporting $\theta_{*}$ optimal, while all the others are better-off reporting truthfully. As a result, the leader receives overall utility $\frac{k}{|\Theta|-1}$.

Conversely, we claim that for any policy $\sigma$, the set of nodes $v$ of which the corresponding types $\theta_{v}$ are incentivized to report $\theta_{*}$ form an independent set, i.e., $\{v\in V:\hat{\theta}_{v}(\sigma)=\theta_{*}\}$ is an independent set. To see this, suppose for a contradiction that types $\theta_{v}$ and $\theta_{v^{\prime}}$ are incentivized to report $\theta_{*}$ but $(v,v^{\prime})\in E$. By our observations above, both $\theta_{v}$ and $\theta_{v^{\prime}}$ will be induced to play $\mathsf{1^{F}}$ and obtain utility $0.5$ each. Thus, we need the following in order for them to have no incentive to report another type.

• (i)

$x_{a_{v^{\prime}}}^{\theta_{v^{\prime}}}=0$, since otherwise the fourth row of the payoff matrix of $\theta_{v}$ would be chosen with some probability, and $\theta_{v}$ would be better-off reporting $\theta_{v^{\prime}}$ to obtain utility strictly greater than $0.5$.

• (ii)

$x_{a_{0}}^{\theta_{v^{\prime}}}=x_{b_{v^{\prime}}}^{\theta_{v^{\prime}}}=0$, and $x_{a_{v^{\prime\prime}}}^{\theta_{v^{\prime}}}=0$ for all $v^{\prime\prime}\in\mathcal{N}(v^{\prime})$, since otherwise one of the first, third and fourth rows of the payoff matrix of $\theta_{v^{\prime}}$ would be chosen with some probability, and $\theta_{v^{\prime}}$ would be better-off reporting truthfully to obtain utility strictly greater than $0.5$.

It follows that, now $\mathbf{x}^{\theta_{v^{\prime}}}$ will pick the last row of the payoff matrix with probability $1$, so $\mathsf{3^{F}}$ becomes the only best response of a type-$\theta_{v^{\prime}}$ player against $\mathbf{x}^{\theta_{v^{\prime}}}$. Thus, we have $o^{\theta_{v^{\prime}}}=\langle\mathbf{x}^{\theta_{v^{\prime}}},\mathsf{3^{F}}\rangle$, in which case, however, $\theta_{v^{\prime}}$ is able to obtain utility $1$ by reporting truthfully, which contradicts the assumption that $\theta_{v^{\prime}}$ is incentivized to report $\theta^{*}$.

Therefore, the number of follower types that are motivated to report $\theta_{*}$ by the optimal policy is exactly to the size of the maximum independent set (and proportional to the overall leader utility). Any $\frac{1}{{(|\Theta|-1)}^{1-\epsilon}}$-approximation algorithm would also provide a $\frac{1}{{|V|}^{1-\epsilon}}$-approximation to the Max-Independent-Set problem, which exists unless P = NP. This completes the proof. ∎

Next, we show that the same inapproximability result for Opt-Policy-IC. Our reduction will still be from Max-Independent-Set but the underlying idea and the instance constructed are different: the previous reduction maps follower types that violate IC to an independent set, which cannot be applied when IC constraints are imposed.

Theorem 3**.**

For any constant $\epsilon>0$, Opt-Policy-IC does not admit any polynomial-time $\frac{1}{{|\Theta|}^{1-\epsilon}}$-approximation algorithm unless P = NP, even when the follower has only three actions.

Proof.

We show a reduction from the Max-Independent-Set problem. Given an instance of Max-Independent-Set specified by a graph $G=(V,E)$, we construct a game with $|V|$ follower types $\Theta=\{\theta_{v}:v\in V\}$; each $\theta_{v}$ corresponds to a node $v$ and appears with probability $\frac{1}{|\Theta|}$. The leader has $2|V|$ actions $\{a_{v}:v\in V\}\cup\{b_{v}:v\in V\}$; and the follower has three actions $\{\mathsf{1^{F}},\mathsf{2^{F}},\mathsf{3^{F}}\}$. The payoffs are given in Figure 4, where $\mathcal{N}(v)$ denotes the set of neighbouring nodes of $v$, and all the empty entries are [math].

Now for any independent set $V^{\prime}$ of size $k$ in $G$, consider a leader policy $\sigma$ with: $o^{\theta_{v}}=\langle a_{v},\mathsf{1^{F}}\rangle$ if $v\in V^{\prime}$, and $o^{\theta_{v}}=\langle b_{v},\mathsf{2^{F}}\rangle$ if $v\notin V^{\prime}$. In this policy, all types find truthful report to be optimal. Hence, the policy is IC and offers the leader overall utility $\frac{k}{|\Theta|}$.

Conversely, for any IC policy $\sigma$, we claim that the set of nodes $v$ of which the corresponding follower types $\theta_{v}$ are motivated to respond with $j=\mathsf{1^{F}}$ form an independent set, i.e., $\{v\in V:o^{\theta_{v}}=\langle i,\mathsf{1^{F}}\rangle\text{ for some }i\}$ is an independent set. Suppose for a contradiction that $\theta_{v}$ and $\theta_{v^{\prime}}$ both respond with $j=\mathsf{1^{F}}$ while $(v,v^{\prime})\in E$. Observe that, when $\theta_{v}$ is reported, $j=\mathsf{1^{F}}$ can be induced only by leader strategy $a_{v}$ because it is strictly dominated by $j=\mathsf{3^{F}}$ in all other cases. Thus, $o^{\theta_{v}}=\langle a_{v},\mathsf{1^{F}}\rangle$ and $o^{\theta_{v^{\prime}}}=\langle a_{v^{\prime}},\mathsf{1^{F}}\rangle$, in which case both $\theta_{v}$ and $\theta_{v^{\prime}}$ obtain utility [math], so $\theta_{v^{\prime}}$ would be better-off reporting $\theta_{v}$, which contradicts the assumption that $\sigma$ is IC.

Therefore, under the optimal IC policy, the number of follower types that respond with $j=\mathsf{1^{F}}$ (which is proportional to the leader’s overall utility) is equal to the size of the maximum independent set in $G$. Any $\frac{1}{{|\Theta|}^{1-\epsilon}}$-approximation algorithm would also provide a $\frac{1}{{|V|}^{1-\epsilon}}$-approximation to the Max-Independent-Set problem. ∎

The bounds in Theorems 2 and 3 are essentially tight as suggested by a polynomial-time approximation algorithm in the next theorem with the matching approximation ratio.

Theorem 4**.**

Both Opt-Policy and Opt-Policy-IC admit a polynomial-time $\frac{1}{|\Theta|}$-approximation algorithm.

Proof.

We show an approximation algorithm for Opt-Policy. The algorithm for Opt-Policy-IC follows a similar procedure.

For each follower type $\theta\in\Theta$, we compute a policy $\sigma_{\theta}$ that maximizes the actual utility the leader obtains on this specific type. We show that (i) $\sigma_{\theta}$ can be computed in polynomial time, and (ii) the best $\sigma_{\theta}$ over all $\theta\in\Theta$ achieves at least $\frac{1}{|\Theta|}$ of the leader utility under the optimal policy.

(i) Polynomial-time computability.

To compute $\sigma_{\theta}$, the idea is to enumerate all possible reporting strategies $\beta\in\Theta$ of type $\theta$ and, for each $\beta$, compute the best policy under the constraint that it is indeed optimal for a type-$\theta$ follower to report $\beta$. An observation that helps simplifying the computation is that we only need to guarantee reporting $\beta$ to be (weakly) preferred by a type-$\theta$ follower to reporting $\theta$, i.e., the truthful report, so it suffices to consider only the two associated outcomes in the leader’s policy, i.e., $o^{\theta}=\langle\mathbf{x}^{\theta},j^{\theta}\rangle$ and $o^{\beta}=\langle\mathbf{x}^{\beta},j^{\beta}\rangle$. This is due to the fact that a policy that prescribes the same strategy $\mathbf{x}^{\theta}$ for all other reports $\beta^{\prime}\notin\{\theta,\beta\}$ will trivially make the truthful report (weakly) preferred to all other reporting strategies $\beta^{\prime}\notin\{\theta,\beta\}$: by reporting truthfully, the follower will already be induced to play his true best response to $\mathbf{x}^{\theta}$. Now, the problem reduces to solving the following LP (liner program), in which $\mathbf{x}^{\theta}$ and $\mathbf{x}^{\beta}$ are the variables, for every pair of possible values of $j^{\theta}$ and $j^{\beta}$. Since $j^{\theta},j^{\beta}\in[n]$, there are $n^{2}$ LPs to be solved; the polynomial-time computability follows immediately.

[TABLE]

Namely, we maximize the leader’s utility obtained on a type-$\theta$ follower under the condition that the follower is incentivized to report $\beta$. Here, (1a) and (1b) guarantee that $j^{\theta}\in\textit{{BR}}_{\theta}(\mathbf{x})$ and $j^{\beta}\in\textit{{BR}}_{\beta}(\mathbf{x})$ as required in the definition of leader policy; and (1c) ensures that reporting $\beta$ is indeed a better choice for a type-$\theta$ follower than reporting $\theta$. The policy $\sigma_{\theta}$ is given by the LP with the best optimal solution among all the $n^{2}$ LPs.

(ii) Utility guarantee.

It remains to show that the best $\sigma_{\theta}$ over all $\theta\in\Theta$ offers the desired utility. For each $\theta$, let $\mu_{\theta}^{\textit{{L}}}$ be the utility offered by $\sigma_{\theta}$. We have $U^{\textit{{L}}}(\sigma_{\theta})\geq\pi\cdot\mu_{\theta}^{\textit{{L}}}$ because $\pi\cdot\mu_{\theta}^{\textit{{L}}}$ is only the part of leader utility obtained on type $\theta$. Now consider an arbitrary feasible policy $\varsigma=\left\{\langle\mathbf{z}^{\theta},k^{\theta}\rangle\right\}_{\theta\in\Theta}$. Let $\hat{\theta}(\varsigma)$ denote a type-$\theta$ follower’s best reporting strategy in response to policy $\varsigma$. It holds that $\mu_{\theta}^{\textit{{L}}}\geq u^{\textit{{L}}}(\mathbf{z}^{\hat{\theta}(\varsigma)},k^{\hat{\theta}(\varsigma)})$ because $\mathbf{x}^{\theta}=\mathbf{z}^{\theta}$ and $\mathbf{x}^{\beta}=\mathbf{z}^{\hat{\theta}(\varsigma)}$ are feasible under (1a)–(1d) when $j^{\theta}$ and $j^{\beta}$ are set to $k^{\theta}$ and $k^{\hat{\theta}(\varsigma)}$, respectively. Let $\theta^{*}=\arg\max_{\theta\in\Theta}\pi_{\theta}\cdot\mu_{\theta}^{\textit{{L}}}$. It follows that, for any feasible $\sigma$,

[TABLE]

Since the choice of $\varsigma$ is arbitrary, $\sigma_{\theta^{*}}$ serves as an $\frac{1}{|\Theta|}$-approximation of the optimal policy. ∎

4.2 An MILP Formulation and the Tractability with Small $\Theta$

Given the hardness result, efficient algorithms for computing the optimal policy seems unlikely. We provide an algorithm based on an MILP as a practical approach to tackle the problem. The MILP formulation also forms the basis of a polynomial-time algorithm for Opt-Policy and Opt-Policy-IC when the size of the follower type set $\Theta$ is fixed.

First, when IC constraints are not imposed, the following program (not linear yet) computes the optimal leader policy, where $\mu_{\theta}^{\textit{{L}}}$, $\mathbf{x}^{\theta}$, $y_{\beta}^{\theta}$, and $p_{j}^{\theta}$ are the variables ($\theta,\beta\in\Theta$, and $j\in[n]$).

[TABLE]

Here, the leader policy is represented by variables $\mathbf{x}^{\theta}$ and $p_{j}^{\theta}$, where $\mathbf{x}^{\theta}$ is the leader mixed strategy prescribed for report $\theta$, and $p_{j}^{\theta}\in\{0,1\}$ captures the induced follower action. Through (2b), it is guaranteed that $p_{j}^{\theta}=1$ only if $j$ is a best response of type $\theta$ to $\mathbf{x}^{\theta}$, and $p_{j}^{\theta}=0$, otherwise. Similarly, $y_{\beta}^{\theta}$ captures the optimal reporting strategy of type $\theta$, i.e., $y_{\beta}^{\theta}=1$ only if reporting $\beta$ is optimal for $\theta$, and this is guaranteed by (2c), in which $M$ is a sufficiently large constant. Finally, each $\mu_{\theta}^{\textit{{L}}}$ captures the utility the leader obtains from true type $\theta$ by (2a).

This program is not yet an MILP because of the quadratic terms $p_{j}^{\theta}\cdot\mathbf{x}^{\theta}$. To linearize the program, we replace these terms with a set of new variables $\tilde{x}_{ji}^{\theta}$, subject to $0\leq\tilde{x}_{ji}^{\theta}\leq x_{i}^{\theta}$ (for all $\theta\in\Theta$, $j\in[n]$, and $i\in[m]$) and $\sum_{i}\tilde{x}_{ji}^{\theta}=p_{j}^{\theta}$ (for all $\theta\in\Theta$ and $i\in[m]$). This guarantees $\tilde{x}_{ji}^{\theta}=p_{j}^{\theta}\cdot{x}_{i}^{\theta}$ given that $p_{j}^{\theta}$ is restricted to be in $\{0,1\}$.

In the situation with IC constraints, we force $y_{\theta}^{\theta}=1$ for all $\theta\in\Theta$ in the above MILP. The “windfall” of the MILP formulation is Theorem 5, the tractability of Opt-Policy and Opt-Policy-IC with a small $\Theta$.

Theorem 5**.**

For a fixed number of follower types, both Opt-Policy and Opt-Policy-IC can be solved in polynomial time.

Proof.

Following Program (2), when $|\Theta|$ is a constant, the size of the joint space of feasible $\mathbf{y}$ and $\mathbf{p}$ is bounded by a polynomial in $n$. We can enumerate every combination in this space and solve Program (2) with $\mathbf{y}$ and $\mathbf{p}$ fixed to values in the combination; the program becomes a linear program as the remaining variables are continuous. ∎

5 Generalization to Mixed Policy

The leader policy we have considered so far consists of a single outcome $\langle\mathbf{x}^{\theta},j^{\theta}\rangle$ for each $\theta$. In this section, we extend the definition of leader policy, and allow a policy to prescribed a distribution of outcomes for each follower type. We call an outcome distribution a mixture and refer to a leader policy consisting of mixtures a mixed policy, as opposed to pure policies in the previous sections.

Now the game proceeds as follows. The leader first commits to a mixed policy. The follower observes the mixed policy and reports a type $\theta$. The leader then samples an outcome $\langle\mathbf{x},j\rangle$ from the mixture prescribed for $\theta$, plays $\mathbf{x}$, and induces the follower to respond with action $j$.

Obviously, the leader utility of an optimal mixed policy is at least that of a pure one as all pure policies are also mixed policies by definition. An immediate question is why further randomization over the outcomes will benefit the leader since the leader is already playing a mixed strategy in every outcome, and randomizing mixed strategies normally will not bring any extra power for a player. It turns out that this extra randomization will be beneficial for the leader when there is follower deception. Intuitively, such randomization can help us tame the incentive constraints for the follower’s type reporting, while the randomization in mixed strategies is responsible for inducing desired follower action responses. The following example provides a more concrete illustration.

Example: Strict Improvement of Mixed Policies

We consider again a security game example with two targets: areas $1$ and $2$. The defender can patrol one of these areas and the poacher chooses one to attack. The poacher has three types $\theta_{*}$, $\theta_{A}$ and $\theta_{B}$ which appear with the equal probability $1/3$. The payoffs matrices are given in the tables below.

Note that the two areas are of equal value to both the defender and poacher type $\theta_{*}$. Thus, $\theta_{*}$ will always attack the less frequently patrolled area, so for an outcome $\langle\mathbf{x},j\rangle$ prescribed for report $\theta_{*}$, it always holds that $x_{j}\leq 1/2$. Suppose without loss of generality that $j=\mathsf{1^{x}}$. Then type $\theta_{A}$ is able to obtain at least $1/2$ by reporting $\theta_{*}$, in which case the actual utility the defender obtains on poacher type $\theta_{A}$, using a pure policy, cannot be more than [math], because for a type-$A$ poacher to obtain utility at least $1/2$, the bottom left entries of the payoff matrices need to be chosen with probability at least $1/2$. The same argument applies to type $\theta_{B}$ if we suppose $j=\mathsf{2^{x}}$. As a result, any pure policy can obtain utility more than [math] on at most one of $\theta_{A}$ and $\theta_{B}$. Further, observe that the true payoff parameters of type $\theta_{*}$ make the game zero-sum. Thus, the actual utility the defender obtains on poacher type $\theta_{*}$ is at most [math] because a type-$\theta_{*}$ poacher is guaranteed utility [math] by reporting truthfully, and when playing against a type-$\theta_{*}$ poacher, the defender always gets the negative of the poacher’s actual utility. As a result, a pure policy obtains positive utility on at most one poacher type; her overall utility is at most $1/3$.

Now we consider a mixed policy as follows:

• •

If type $\theta_{*}$ is reported, choose outcomes $\langle(1/2,1/2),\mathsf{1^{x}}\rangle$ and $\langle(1/2,1/2),\mathsf{2^{x}}\rangle$, each with probability $1/2$;

• •

If type $\theta_{A}$ is reported, choose outcome $\langle(1,0),\mathsf{1^{x}}\rangle$ with probability $1$; and if $\theta_{B}$, choose $\langle(0,1),\mathsf{2^{x}}\rangle$ with probability $1$.

As such, types $\theta_{A}$ and $\theta_{B}$ can only obtain $-1/2$ in expectation if they report $\theta_{*}$. This incentivizes both of them to report truthfully, yielding expected defender utility $2/3$ — an improvement of at least $1/3$ compared to the optimal pure policy.

5.1 Mixtures with Support Size $n$ Suffice

To compute the optimal mixed policy, one challenge is that a mixture may be supported on an infinite set since there are infinitely many mixed strategies and hence, as many outcomes. Fortunately, using a revelation-principle-type argument, we prove that it suffices to consider mixtures supported on at most $n$ outcomes; each outcome induces the follower to choose a distinct action in $[n]$. This is shown in Proposition 6 below.

Proposition 6**.**

For any feasible mixture $\phi$ prescribed for a reported type $\theta\in\Theta$, there is a feasible mixture with support size $\tilde{n}\leq n$ that yields the same utility as does $\phi$, for both the leader and the follower.

Proof.

We will show that for any given mixture $\phi$ prescribed for report $\theta$ we can merge outcomes that induce the same follower response into one outcome, and this will not change both players’ expected utility.

Formally, let $\mathcal{O}$ be the support set of $\phi$; let $p_{\langle\mathbf{x},j\rangle}>0$ be the probability assigned to outcome $\langle\mathbf{x},j\rangle\in\mathcal{O}$. For each $j\in[n]$, let $\mathcal{O}_{j}=\{\langle\mathbf{x},j^{\prime}\rangle\in\mathcal{O}:j^{\prime}=j\}$ be the set of outcomes that induce follower action $j$. Let $\tilde{p}_{j}=\sum_{\langle\mathbf{x},j\rangle\in\mathcal{O}_{j}}p_{\langle\mathbf{x},j\rangle}$ and $\tilde{\mathbf{x}}_{j}=\frac{1}{\tilde{p}_{j}}\sum_{\langle\mathbf{x},j\rangle\in\mathcal{O}_{j}}p_{\langle\mathbf{x},j\rangle}\cdot\mathbf{x}$. Let $\tilde{\phi}$ be a mixture supported on $\{\langle\tilde{\mathbf{x}}_{1},1\rangle,\dots,\langle\tilde{\mathbf{x}}_{n},n\rangle\}$, with probability $\tilde{p}_{j}$ for each $\langle\tilde{\mathbf{x}}_{j},j\rangle$. This new mixture will not change the leader’s utility when $\theta$ is reported because of the following:

[TABLE]

By changing labels of the utility function, we can obtain $\sum_{\langle\mathbf{x},j\rangle\in\mathcal{O}}p_{\langle\mathbf{x},j\rangle}\cdot u_{\beta}^{\textit{{F}}}(\mathbf{x},j)=\sum_{j\in[n]}\tilde{p}_{j}\cdot u_{\beta}^{\textit{{F}}}(\tilde{\mathbf{x}}_{j},\ j)$ for any follower type $\beta\in\Theta$, so $\tilde{\phi}$ will not change the follower’s utility, either, irrespective of his true type.

It remains to show that $\tilde{\phi}$ is indeed a valid mixture. Clearly, $\tilde{p}_{j}\geq 0$ and $\sum_{j}\tilde{p}_{j}=\sum_{j}\sum_{\langle\mathbf{x},j\rangle\in\mathcal{O}_{j}}p_{\langle\mathbf{x},j\rangle}=\sum_{\langle\mathbf{x},j\rangle\in\mathcal{O}}p_{\langle\mathbf{x},j\rangle}=1$, so $\tilde{p}_{j}$ is a valid probability distribution. In addition, we need each outcome $\langle\tilde{\mathbf{x}}_{j},j\rangle$ to be feasible, i.e., $j\in\textit{{BR}}_{\theta}(\tilde{\mathbf{x}}_{j})$, so that $j$ can indeed be induced by $\tilde{\mathbf{x}}_{j}$. This is true because the set $\mathcal{X}_{j}=\{\mathbf{x}\in\Delta_{m}:j\in\textit{{BR}}_{\theta}(\mathbf{x})\}=\{\mathbf{x}\in\Delta_{m}:u_{\theta}^{\textit{{F}}}(\mathbf{x},j)\geq u_{\theta}^{\textit{{F}}}(\mathbf{x},j^{\prime})\text{ for all }j^{\prime}\in[n]\}$ is a convex set, and by definition, all $\mathbf{x}$ such that $\langle\mathbf{x},j\rangle\in\mathcal{O}_{j}$ are in $\mathcal{X}_{j}$. Thus, $\tilde{\mathbf{x}}$, as a convex combination of $\mathbf{x}\in\mathcal{X}_{j}$, also lies in $\mathcal{X}_{j}$, so $j\in\textit{{BR}}_{\theta}(\tilde{\mathbf{x}}_{j})$. Therefore, $\langle\tilde{\mathbf{x}}_{j},j\rangle$ is a feasible outcome and $\tilde{\phi}$ a valid mixture. This completes the proof. ∎

Following this result, we can represent the mixture for each type $\theta$ as a vector $(p_{1}^{\theta},\dots,p_{n}^{\theta})\in\Delta_{n}$, along with a support set $\{\mathbf{x}_{1}^{\theta},\dots,\mathbf{x}_{n}^{\theta}\}$ of $n$ mixed strategies; the mixture samples each outcome $\langle\mathbf{x}_{j}^{\theta},j\rangle$ with probability $p_{j}^{\theta}$. Note that this representation allows outcomes involved to be invalid because it is possible that some follower action $j$ cannot be induced by any leader strategy, i.e., $j\notin\textit{{BR}}_{\theta}(\mathbf{x})$ for all $\mathbf{x}\in\Delta_{m}$. Thus, a mixture is valid only if $j\in\textit{{BR}}_{\theta}(\mathbf{x}_{j}^{\theta})$ for all $j$ such that $p_{j}^{\theta}>0$.

5.2 Computing the Optimal Mixed Policy

Proposition 6 implies that there exists an optimal mixed policy of polynomial size. Nevertheless, out next theorem, Theorem 7, shows that the problem of computing the optimal mixed policy, referred to as Opt-XPolicy, remains inapproximable in the situation without IC constraints. Interestingly, after we impose IC constraints, the problem Opt-XPolicy-IC becomes tractable. We present these results next.

Theorem 7**.**

For any constant $\epsilon>0$, Opt-XPolicy does not admit any polynomial-time $\frac{1}{{(|\Theta|-1)}^{1-\epsilon}}$-approximate algorithm unless P = NP, even when the follower has only three actions.

Proof.

We show a reduction from the same Max-Independent-Set problem as in the proof of Theorem 2, but we will need to adapt the correctness proof to mixed policies here.

For one direction of the reduction, if there is an independent set $V^{\prime}$ of size $k$ in $G$, the same policy we constructed in the proof of Theorem 2 achieves overall leader utility $\frac{k}{|\Theta|-1}$ for the same argument.

For the other direction, observe that it holds still with mixed policies that the leader gets $1$ when $\theta_{*}$ is reported, and [math], otherwise; hence, the leader’s overall utility is proportional to the number of types $\theta_{v}$, $v\in V$, who report $\theta_{*}$. We show that, for any mixed policy $\sigma$, $\{v\in V:\hat{\theta}_{v}(\sigma)=\theta_{*}\}$ is an independent set to conclude the proof. Suppose for a contradiction that $\hat{\theta}_{v}(\sigma)=\theta_{*}$ and $\hat{\theta}_{v^{\prime}}(\sigma)=\theta_{*}$ but $(v,v^{\prime})\in E$. We make the following observations:

• (i)

$p_{\mathsf{1^{x}}}^{\theta_{v}}=0$ because $\mathsf{1^{x}}$ is strictly dominated by $\mathsf{2^{x}}$ for a follower of type $\theta_{v}$ and hence $\mathsf{1^{x}}\notin\textit{{BR}}_{\theta_{v}}(\mathbf{x})$ for any leader strategy $\mathbf{x}$. It follows that we need $p_{\mathsf{2^{x}}}^{\theta_{v}}>0$ or $p_{\mathsf{3^{x}}}^{\theta_{v}}>0$, as $(p_{\mathsf{1^{x}}}^{\theta_{v}},p_{\mathsf{2^{x}}}^{\theta_{v}},p_{\mathsf{3^{x}}}^{\theta_{v}})\in\Delta_{3}$.

• (ii)

Since a follower of type $\theta_{v^{\prime}}$ gets utility $0.5$ by reporting $\theta_{*}$, we need $p_{\mathsf{2^{x}}}^{\theta_{v}}\cdot x_{\mathsf{2^{x}},a_{v}}^{\theta_{v}}=0$ and $p_{\mathsf{3^{x}}}^{\theta_{v}}\cdot x_{\mathsf{3^{x}},a_{v}}^{\theta_{v}}=0$ as otherwise the action profile $\langle a_{v},\mathsf{2^{x}}\rangle$ would be chosen with some probability and $\theta_{v^{\prime}}$ would be better-off reporting $\theta_{v}$ to get more than $0.5$.

Now, consider the two possible cases implied by (i).

• •

If $p_{\mathsf{3^{x}}}^{\theta_{v}}>0$, we then have $x_{\mathsf{3^{x}},a_{v}}^{\theta_{v}}=0$ by (ii), which means some action profile $\langle i,\mathsf{3^{x}}\rangle$, $i\neq a_{v}$, would be chosen with some probability, in which case a follower of type $\theta_{v}$ would be better-off reporting truthfully instead of reporting $\theta_{*}$ — a contradiction.

• •

If $p_{\mathsf{2^{x}}}^{\theta_{v}}>0$, we have $x_{\mathsf{2^{x}},a_{v}}^{\theta_{v}}=0$ by (ii). Now in order for a type-$\theta_{v}$ follower to still find reporting $\theta_{*}$ optimal, we need $x_{\mathsf{2^{x}},i}^{\theta_{v}}=0$ for all $i\in\{a_{0},b_{v}\}\cup\mathcal{N}({v})$ (correspondingly, the first, third and fourth rows of the payoff matrix) as otherwise truthful report would gain type $\theta_{v}$ a higher utility; consequently, only the fifth row of the payoff matrix will be chosen by $\mathbf{x}_{\mathsf{2^{x}}}^{\theta_{v}}$, in which case, however, $\textit{{BR}}_{\theta_{v}}(\mathbf{x}_{\mathsf{2^{x}}}^{\theta_{v}})=\{\mathsf{3^{x}}\}$, so $\langle\mathbf{x}_{\mathsf{2^{x}}}^{\theta_{v}},\mathsf{2^{x}}\rangle$ cannot be a valid outcome — a contradiction to the assumption that $p_{\mathsf{2^{x}}}^{\theta_{v}}>0$.

This completes the proof. ∎

The bound in the above inapproximability result is tight following a similar argument for Opt-Policy. Moreover, an MILP formulation for Opt-XPolicy can be obtained by modifying Program (2), and this yields a polynomial-time algorithm for Opt-XPolicy when the number of follower types is fixed. If we further impose IC constraints, the constraints remove the only integer variables $y_{\beta}^{\theta}$ in the MILP (variables $p_{j}^{\theta}$ are relaxed now) and result in a linear program; this establishes the tractability of Opt-XPolicy-IC. We sketch the results in Theorems 8–10 below.

Theorem 8**.**

Opt-XPolicy* admits a polynomial-time $\frac{1}{|\Theta|}$-approximation algorithm.*

Proof Sketch.

The proof follows from a similar argument for Theorem 4. ∎

Theorem 9**.**

For a fixed number of follower types, Opt-XPolicy can be solved in polynomial time.

Proof Sketch.

We modify Program (2) for a formulation for Opt-XPolicy: replace every $\mathbf{x}^{\theta}$ by $\mathbf{x}_{j}^{\theta}$ wherever they are associated with $p_{j}^{\theta}$; and relax every $p_{j}^{\theta}$ to be in $[0,1]$. The modified program can be linearized into an MILP in the same way we linearize Program (2). By the same argument for Theorem 9, the MILP formulation yields a polynomial-time algorithm for Opt-XPolicy when the number of follower types is fixed. ∎

Theorem 10**.**

Opt-XPolicy-IC* can be computed in polynomial time.*

Proof Sketch.

The problem can be formulated as a linear program as follows, where $\tilde{\mathbf{x}}_{j}^{\theta}$ and $p_{j}^{\theta}$ are the variables ($j\in[n],\theta\in\Theta$).

[TABLE]

The strategies $\mathbf{x}_{j}^{\theta}$ in the leader policy can be extracted from the solution as $\mathbf{x}_{j}^{\theta}=\frac{1}{p_{j}^{\theta}}\cdot\tilde{\mathbf{x}}_{j}^{\theta}$. ∎

6 Empirical Evaluation

We evaluate our framework with games generated randomly using the well-known covariance game model Nudelman et al. (2004). We generate the players’ payoffs uniformly at random from the range $[0,1]$, and then adjust the follower’s payoffs by blending them with the negative value of the leader’s payoffs; the degree of blending is controlled by a parameter $\alpha\in[0,1]$, i.e., $u_{\theta}^{\textit{{F}}}\leftarrow(1-\alpha)\cdot u_{\theta}^{\textit{{F}}}-\alpha\cdot u^{\textit{{L}}}$. As such, the game is zero-sum when $\alpha=1$; and, when $\alpha=0$, payoffs of the leader and the follower are completely uncorrelated. We also generate the probabilities $\pi_{\theta}$’s uniformly at random.

We compare the leader’s utility obtained by different policies/approaches proposed in this paper and previous research, including: (1) Optimal pure policy (labeled Opt), with and without IC; (2) Optimal mixed policy (OptX), with and without IC; (3) Bayesian Stackelberg Equilibrium (BSE); (4) Optimal leader strategy when the follower truthfully reports his type (Truthful), and (5) the same strategy but when the follower strategically reports his type (Deceitful).

Figure 5 depicts our results. The first three figures, (a)–(c), show the variance of leader utility with $\alpha$ under different numbers of player actions. For ease of comparison, the results are depicted as ratios to the leader utility in the truthful situation. The figures exhibit quite similar patterns. All our proposed polices (lines in blue) improve the leader’s utility significantly upon BSE. The utilities obtained are also very close to the truthful utility even when IC constraints are imposed. When IC is not required, the optimal policies even perform better in almost all experiments. Surprisingly, even when the leader naïvely trusts a deceitful follower, the results are very positive; the utility obtained is usually very close to the truthful utility. This is in contrast to our theoretical examples. An explanation is that in randomly generated games, the negative effect of follower deception is likely to be cancelled out by positive effect brought by benign follower types: as we have observed in Section 3, follower deception is not always bad for the leader.

The utility differences are the most significant when $\alpha$ is around $0.5$. Intuitively, this is because when $\alpha$ approaches $1$ the diminishing uncertainty over follower types reduces the effect of follower deception. But when $\alpha$ is close to [math], payoffs of the leader and the follower are less correlated and their objectives less conflicted; hence, when the follower leverages deception to increase his utility, there is a lower chance that this will decrease the leader’s utility.

We further investigate the effects of varying numbers of actions and of follower types. From Figure 5 (d) we see that utility differences (except for BSE) quickly diminish when the number of actions increase. Similar to the situation when $\alpha$ is close to [math], the level of randomness of the payoffs increases as the action space grows, which dissolves the negative effect of follower deception. In contrast, as shown in Figure 5 (e), the effect of follower deception appear to increase with the number of follower types as leader utility in the deceitful setting drops. Utility offered by the optimal mixed IC policy, however, decreases much slower. Hence, the optimal mixed IC policy offers a scalable practical solution when there is a large number of follower types. At a high level, these results suggest that uncertainty over follower’s payoff information is a major amplifier of the negative effect of follower deception.

To explore situations in which follower deception is likely to be more harmful, we slightly modify the covariance game model, generating the follower type set with multiple blending parameters. In Figure 5 (f), the results are obtained on a follower type set in which half of the types are generated with $\alpha=0.5$, and another half generated with $\alpha^{\prime}$ as on the x-axis. Interestingly, the difference between the approaches suddenly becomes more distinguishable, and there is a clear gap between the optimal and the deceptive situations. An in-depth characterization of when deception tends to be more harmful to the leader and when it is not would be an interesting question for future work.

Robustness to Tie-breaking

In our model, we assumed that the follower will break ties in favor of the leader. The assumption is already widely adopted in the literature of Stackelberg games because such favorable tie-breaking behavior can typically be induced by an infinitesimal deviation of the leader’s strategy, though in some rare and extreme cases such inducibility may not hold true (see a recent discussion in Guo et al. (2018)). Our next set of simulation results experimentally demonstrate that inducibility is not an issue in our model for naturally generated instances. In the simulations, we enforce a small utility gap of at least $\epsilon$ for the follower between taking the induced action and any other action (the same with the induced reporting strategy). This removes all the ties, and makes the follower strictly prefer the action (by $\epsilon$) the leader intends to induce. The results are depicted in Table 1, with $\epsilon$ varying from $10^{-5}$ to $10^{-1}$. The experimental results show that our solution is robust to tie breaking and always maintains optimality for $\epsilon\leq 10^{-3}$ (recall that the follower’s payoffs are in $[-1,1]$). Moreover, desired follower actions are almost always inducible only except for a small number of instances when $\epsilon=0.1$.444Note that we list $\epsilon=0.1$ only for illustrative purpose as in practice one would never want to use such a big $\epsilon$ to induce tie breaking. There are extended approaches able to generate feasible robust solutions for every $\epsilon\in[0,1]$ (see Pita et al. (2010)), but this is not the focus of this paper. These results empirically show that the issue of tie-breaking does not affect the validity of our approach in natural instances.

7 Conclusion

In this paper, we point out potential occurrences of (imitative) follower deception in Stackelberg games and the risk when they are ignored. We then propose a framework to design leader policies against such deception and provide a systematic study of the computational aspects of this framework. Our results shows that handling follower deception is hard in general, even when the searching of optimal policy is expanded into the less complicated space of mixed policies. Our experiments indicate that the loss due to ignoring follower deception grows with uncertainty over follower types. However, when there is only a small number of types, the average loss appear to be very small compared to results in the theoretical analysis.

There are a number of potential directions for future work. Perhaps the first ones to investigate are several extended settings, including the setting with no prior knowledge about the distribution of follower types, and the setting where follower types fall into a continuous space (e.g., when each payoff parameter falls in an interval) instead of a finite set as in our current model. For the former, the worst-case analysis might be relevant, and an even more challenge task arises when the leader is also trying to learn the prior distribution of follower types from the interaction. For the latter, even the follower’s problem of how to optimally misreport his type does not appear to have an obvious solution, unlike in our current model (where the follower simply needs to enumerate all types in the finite type set). Another natural setting is that the follower does not need to keep imitating the fake type, switching back to playing their actual best response after the leader finishes learning. This is yet another generalization of the standard Bayesian Stackelberg game model, and the preliminary thought is that the same policy-based approach is also able to guarantee the leader as least her utility in the Bayesian Stackelberg equilibrium (similar to our Proposition 1).

Besides focusing on normal-form Stackelberg game models, it would also be interesting to position this work in a specific application area, such as security games where follower deception might be more common, yet more harmful, because of the adversarial nature of the game.