Designing for Emergent Security in Heterogeneous Human-Machine Teams

TL;DR

This paper explores how autonomous machines and human decisionmakers can collaboratively improve system security against attacks through scheduling policies, revealing conditions that enhance or hinder their influence.

Contribution

It introduces a simple scheduling policy demonstrating how machine autonomy can enhance security, and identifies constraints that limit machine influence over human decisionmakers.

Findings

Increasing machine scheduling penetration generally improves security.

Unfavorable server availability constraints can nullify machine influence.

Coordination among machines does not guarantee influence if constraints are restrictive.

Abstract

This work seeks to design decisionmaking rules for autonomous agents to jointly influence and optimize the behavior of teamed human decisionmakers in the presence of an adversary. We study a situation in which computational jobs are scheduled on servers by a collection of autonomous machines in concert with self-interested human decisionmakers, and the human and machine schedulers must react to an adversary's attack on one of the servers. We show a simple machine scheduling policy such that if all schedulers have permission to schedule jobs on all servers, increasing the penetration of machine schedulers always increases the level of security in the system, even when the machine schedulers have no explicit coordination or communication amongst themselves. However, we show a companion result in which simple constraints on server availability can nullify the machine schedulers' ability to effectively influence human schedulers; here, even if machine schedulers control an overwhelming majority of jobs, are socially-aware, and fully coordinated amongst themselves, they are incapable of influencing human decisionmakers to mitigate the harm of an attack.

Equations

$$\mathcal{C}(x):=\sum _{i=1}^n\frac{x_i{\tau }_i(x_i)}{n}.$$

$${\mathcal{C}}^{\alpha }(x)$$

$${\bar{x}}^k\in \underset{x^k\in r_k\Delta (n)}{\mathrm{arginf}}{\mathcal{C}}^{\alpha }\left({\bar{x}}^{\mathrm{s}},x^k,{\bar{x}}^{-k}\right)$$

$${\bar{x}}_i^{\mathrm{s}}>0$$

$${\mathcal{C}}^{\alpha }(\bar{x}(\alpha ))={\mathcal{C}}^{\alpha }(x^{\ast }(\alpha )).$$

$${\mathcal{C}}^{\alpha }(\bar{x}(\alpha ))\le {\mathcal{C}}^{\alpha }\left(x^{\mathrm{b}}\right).$$

$${\mathcal{C}}^{\alpha }\left(\bar{x}(\alpha )\right)\le {\mathcal{C}}^{\alpha }\left(\bar{x}’(\alpha )\right).$$

$${\mathcal{C}}^{\alpha }\left(\bar{x}(r,\alpha )\right)=\underset{x\in n\Delta (n)}{\inf }{\mathcal{C}}^{\alpha }(x).$$

$$\bar{r}$$

$${\mathcal{C}}^{\alpha }(\bar{x})=\min \left\{\frac{n}{n-1},1+\frac{\alpha }{n}-\frac{{\alpha }^2(n-1)}{4n^2}\right\}.$$

$${\mathcal{C}}^{\alpha }(\bar{x})=\min \left\{\frac{n}{n-1},\frac{r^2+r\left(\frac{\alpha (n-1)}{n}-2\right)+n}{n-1}\right\}$$

$${\mathcal{C}}^{\alpha }(\bar{x})$$

$$\le {\mathcal{C}}^{\alpha }\left(x^{\mathrm{b}}\right)=1+\frac{\alpha }{n}.$$

$${\mathcal{C}}^{\alpha }(\bar{x}(\alpha ))={\mathcal{C}}^{\alpha }(x^{\mathrm{n}}(\alpha ))=\min \left\{\frac{n}{n-1},1+\frac{\alpha }{n}\right\}.$$

$$J(x^{\mathrm{m}},x^{\mathrm{s}})={\mathcal{C}}^{\alpha }\left(x^{\mathrm{m}}+x^{\mathrm{s}}\right).$$

$$x_1^{\mathrm{s}}>0$$

$$x_2^{\mathrm{s}}>0$$

$${\hat{x}}^{\mathrm{m}}(\alpha )$$

$${\hat{x}}^{\mathrm{s}}(\alpha )$$

$$x_{2}^{\rm m}=\left\{\begin{array}[]{ll}1-\frac{\alpha(n-2)}{2n}&\mbox{if }\alpha<\frac{2n}{n-2}\left(2-\sqrt{\frac{2n}{n-1}}\right),\\ \frac{1}{n-1}&\mbox{otherwise. }\end{array}\right.$$

$${\mathcal{C}}^{\alpha }(\hat{x}(\alpha ))=\min \left\{\frac{n}{n-1},1+\frac{\alpha }{n}-\frac{{\alpha }^2(n-2)}{8n^2}\right\}.$$

$$\text{vspace}-1mm{\mathcal{C}}^{\alpha }(\bar{x}(\alpha ))\le {\mathcal{C}}^{\alpha }(x^{\mathrm{n}}(\alpha )),$$

$$\text{vspace}-1mmk\in \underset{j\in \{1,\dots ,n\}}{\mathrm{argmin}}{\tau }_j(x_j^{\ast }).$$

$$\text{vspace}-1mm{\tau }_k(x_k^{\ast })\le {\tau }_j(x_j^{\ast }).$$

$${\bar{x}}_i={\bar{x}}_j$$

$$\text{vspace}-1mm{x}_1+\alpha \ge x_2.$$

$$\text{vspace}-1mm{\mathcal{C}}^{\alpha }(x)\ge 1+\frac{\alpha }{n},$$

$$\underset{x}{\text{mbox}minimize}$$

$$\sum _{i=1}^n{x}_i=n,$$

$$x_1+\alpha \ge x_2,$$

$$x_1\ge 0.$$

$$2x_1+\alpha -\lambda +{\mu }_1+{\mu }_2$$

Full text

I Introduction

The recent rush of technology into society’s daily life has promised to solve grand problems, but also may bring with it grand new risks. New technological paradigms such as the Internet of Things allows people to interact with their devices in unprecedented ways, but increased means of interaction may increase vulnerability to adversarial manipulation [1, 2]. Increasingly, society and technology are participants in each other’s affairs, which requires that engineers and computer scientists must be increasingly aware of the effects of each on the other. Autonomous decisionmakers must be designed to interact well with human decisionmakers, and this may look fundamentally different from how autonomous decisionmakers typically interact with one another [3].

In response to these new challenges, game theory (the formal study of interactive decisionmaking) has emerged as a set of mathematical tools which promise to shed some light on the central design tradeoffs in this space [4, 5, 6]. In applying game theory to these types of systems, it is common to speak of “emergent behavior” in distributed decisionmaking; that is, game theory studies the aggregate behavior which emerges from the entangled decisionmaking processes of autonomous machine agents, human users, and strategic attackers [7, 8, 9].

This paper represents an initial study on a concept that we term emergent security. That is, we seek fundamental principles which will allow the principled design of autonomous decisionmakers in sociotechnical systems; these autonomous agents should react to the actions of attackers in an intelligent, strategically-aware way to mitigate the harm of attacks.

A strategic attacker’s approach may be twofold: first, the attacker may directly attack the system and reduce quality-of-service [10]. Alternatively, the attacker may indirectly reduce quality-of-service by influencing ordinary system users to engage in inefficient behavior; this can include attack methods such as manipulative “social-engineering” attacks [11, 12].

The defending autonomous agents’ approach is similarly twofold: first, agents should be designed to directly avoid and mitigate the effects of the attacker’s actions; second, the autonomous agents should be designed to influence the behavior of ordinary system users so that they avoid and mitigate the effects of the attacker’s actions [13, 14, 15].

This paper asks whether and under what circumstances heterogeneous teams of human and machine decisionmakers can act in concert to provide security for distributed systems, despite a lack of centralized coordination among the decisionmakers. Specifically, we seek distributed decisionmaking architectures that provide emergent security; that is, the distributed decisionmaking rules and interaction framework guarantee that aggregate emergent behavior in the system responds effectively to adversarial manipulation.

We pose a model of collaborative job scheduling on servers: there is a collection of $n$ servers and a constant inflow of jobs; each job requires scheduling on any of the servers. Each job is given either to one of a collection of “machine schedulers” (which then select a server for the job based on a pre-specified algorithm) or the job is self-scheduled (in which case the job is assigned to a server with minimum current service time).

We consider a situation in which an intelligent attacker, wishing to decrease system throughput, selects a server and directly degrades its performance (e.g., by performing a denial-of-service attack that increases average service time) [16]. Subsequently, the machine schedulers and self-scheduled jobs may react to the increased service time on the attacked server and modify their scheduling choices accordingly. The attacker’s hope is to maximize the resulting performance degradation, both directly due to increased service times on the affected server and indirectly by manipulating the human and machine scheduling policies to create a cascade of inefficiencies affecting other servers.

In light of this model, our question is simple: can the decision rules of the machine schedulers be designed in such a way that system security emerges as the result of the interactive decisionmaking of the various schedulers? We say that a system exhibits strong emergent security if, for all attacks, the schedulers collectively respond optimally to the attack. On the other hand, we say that a system exhibits weak emergent security if, for all attacks, the performance of the schedulers’ collective response is at least no worse than a fixed scheduling policy. Given these definitions, our results are threefold:

Symmetric scheduling: When every server is available to every scheduler and the machine schedulers apply a simple locally-optimal scheduling policy, every system exhibits strong emergent security provided that the fraction of jobs controlled by machine schedulers exceeds some threshold (where this threshold is strictly less than $1$). Crucially, this threshold is independent of the number of machine schedulers.

Symmetric scheduling and linear server delay: Here, if the $n$ servers each have identical linear delay functions, provided that at least $1/n$ of the jobs are controlled by machine schedulers, the collective response to any attack is optimal (i.e., the system has strong emergent security even if only a small minority of jobs are machine-controlled). Furthermore, every system with linear server delay functions exhibits weak emergent security. That is, regardless of the mass of jobs controlled by machine schedulers, we have that for any attack, collective scheduling behavior is always at least as good as a fixed scheduling policy that is unresponsive to attacks.

Constrained server availability: However, the above results need not hold in the case that each scheduler can only access a subset of servers. In this case, even if the majority of jobs are centrally controlled by a single machine scheduler, the system may perform nearly as poorly as a completely unresponsive policy. Here, though the system still exhibits weak emergent security, the ability of the machine schedulers to reduce system cost is severely curtailed.

Our results suggest that emergent security of heterogeneous human-machine teams is a valid goal and can be attained with simple decision rules. However, designers must take care to understand how constraints and heterogeneity can subvert an otherwise performant system.

II Model

We are given a set of $n$ servers and a total load of $n$ units of jobs requiring service. Following from [17], each individual job is taken to be infinitesimally “small,” so that each job contributes a negligible amount to congestion. We denote the load (or mass of scheduled jobs) at server $i$ by $x_{i}\geq 0$, and $\tau_{i}(x_{i})$ denotes the delay suffered by a job at server $i$ under load $x_{i}$. We adopt the usual convention that each $\tau_{i}(x_{i})$ is convex, nondecreasing, and continuously differentiable, and assume that the uncongested service time at all servers is equal, or for all $i,j$, $\tau_{i}(0)=\tau_{j}(0)$. A service profile is denoted $x:=(x_{1},x_{2},\dots,x_{n})$; we require that all jobs are serviced so that $\sum_{i}x_{i}=n$, which we denote by $x\in n\Delta(n)$111 Note that for $a>0$, we write $a\Delta(n)$ to denote the standard probability simplex with each of its constraints multiplied by $a$, so that its vertices are of the form $(0,\dots,a\dots,0)$. . Our key cost metric is average service delay, denoted

[TABLE]

II-A Adversarial action

An adversary, wishing to disrupt the system and increase system cost, selects a server (throughout this paper, we adopt the convention that this server is numbered $1$) and attacks it with strength $\alpha>0$. The effect of this attack is to uniformly increase service times222Here, we consider an additive degradation of service. Naturally, this degradation could take other forms. We adopt this particular formulation for simplicity, postponing the study of other formulations for future work. at server $1$ by an amount $\alpha$; this is modeled by replacing the nominal delay function $\tau_{1}(x_{1})$ with the degraded delay function $\tau_{1}^{\alpha}(x_{1}):=\tau_{1}(x_{1})+\alpha$. For convenience, for any $i\geq 1$, we write $\tau_{i}^{\alpha}(x_{i})=\tau_{i}(x_{i})$.

Since the attacker’s action directly increases service times for any job on server $1$, it induces a new system cost of

[TABLE]

II-B Heterogeneous Human-Machine Teamed Scheduling

The main goal of this paper is to understand the effects of human-machine teaming on the security of distributed systems. To accomplish this, we consider a case in which there are $m$ engineered schedulers (called machine schedulers) acting in conjunction with a population of uncoordinated human schedulers. The system designer endows the machine schedulers with decision rules with which they select a scheduling profile, whereas the human schedulers select a server merely on the basis of minimizing their own service delay.

Machine scheduler $k\in\{1,\dots,m\}$ is responsible for scheduling a mass of $r_{k}$ jobs, with $r:=\sum_{k=1}^{m}r_{k}\leq n$. We refer to $r$ as the machine penetration level of the system. We term the remaining $n-r$ jobs selfish jobs, and we model their “behavior” by assuming that each infinitesimal job among the selfish jobs has a human owner that schedules it with the singular goal of minimizing its service time, given the attacker’s action and the choices of other jobs.

Figure 1 contains a simple depiction of the model. We allow each machine scheduler to schedule on any of the $n$ servers, so each machine scheduler $k$ selects a service profile $x^{k}:=(x_{1}^{k},\dots,x_{n}^{k})\in x^{k}\in r_{k}\Delta(n)$, satisfying $x^{k}_{i}\geq 0$ and $\sum_{i=1}^{n}x^{k}_{i}=r_{k}$. Given a collection of machine-scheduled profiles $\{x^{k}\}$ and a selfishly-scheduled profile $x^{\rm s}$, the aggregate load on machine $i$ is given by $x_{i}=\sum_{k=1}^{m}x_{i}^{k}+x_{i}^{\rm s}$. The aggregate scheduling profile resulting from teamed scheduling is represented $x=(x^{\rm s},x^{1},\dots x^{m})$. Sometimes, to highlight the dependence of a scheduling profile on the actions of a particular machine $k$, we may also represent an aggregate profile as $x=(x^{\rm s},x^{k},x^{-k})$, where $x^{-k}$ denotes the scheduling by machine schedulers other than $k$.

We model the aggregate behavior resulting from the collective decisionmaking of the $m$ machine schedulers and the selfish jobs by a concept that we term team equilibrium $\bar{x}$, or an aggregate scheduling profile satisfying

[TABLE]

and

[TABLE]

Note that at a team equilibrium, (3) implies that each machine scheduler is scheduling (globally) optimally (given the choices of others), and (4) implies that each selfish job is selecting a server with minimum delay (given the choices of others). While equilibrium existence in arbitrary games is not guaranteed, known results for heterogeneous congestion games give us the following proposition:

Proposition II.1

For any $\alpha\geq 0$, any $n\geq 2$, any $m\geq 1$, and any $\{r^{k}\}_{k=1}^{m}$ satisfying $\sum_{k=1}^{m}r^{k}\leq n$, the above game has a team equilibrium $\bar{x}$.

This follows immediately from elementary equilibrium existence results in nonatomic games [18, 19, 20].

II-C Emergent Security

We say that a system exhibits emergent security if the teamed agents in the system (both human and engineered) naturally adapt to attacks in such a way that the system performance remains near-optimal even when the system is under attack. Since the team equilibrium concept implicitly assumes no coordination between agents, it is a natural means to study the emergent security arising from various decision rules.

In this context, we distinguish between different degrees of security by comparing the performance of a team equilibrium against that of two benchmarks:

Optimal scheduling

Given $\alpha$, we denote an optimal scheduling policy by $x^{*}(\alpha)\in\operatorname*{arg\,min}_{x}\mathcal{C}^{\alpha}(x)$. The ultimate goal in the emergent security paradigm is for the performance of team equilibria to equal that of $x^{*}(\alpha)$.

Unresponsive scheduling

Here, we compare the performance of a team equilibrium to that of an unresponsive baseline scheduling policy that we denote $x^{\rm b}(\alpha)$; this policy is optimal at no-attacks when $\alpha=0$, but is unresponsive to changes in $\alpha$. That is, for all $\alpha\geq 0$, $x^{\rm b}(\alpha)=x^{*}(0)$. Since this scheduling policy is constant in $\alpha$, we typically denote it simply $x^{\rm b}$.

Using these two definitions, we can now state the main qualitative performance metric of the paper:

Definition II.1

We say that a system exhibits strong emergent security if, for all $\alpha\geq 0$, it holds that all team equilibria are optimal. That is, if $\bar{x}(\alpha)$ is a team equilibrium, we have that for all $\alpha\geq 0$,

[TABLE]

On the other hand, we say that a system exhibits weak emergent security if, for all $\alpha\geq 0$, it holds that all team equilibria are no worse than unresponsive scheduling. That is, if $\bar{x}(\alpha)$ is a team equilibrium, we have that

[TABLE]

In each of our forthcoming contributions, we exhibit a system setting and then determine which of Definition II.1’s properties hold.

III Contributions

III-A Team scheduling exhibits emergent security for symmetric schedulers

Our first theorem is a general result on the effectiveness of heterogeneous human-machine teams; it establishes a baseline guarantee that performance guarantees for such systems are monotone nondecreasing in the penetration of machine schedulers. Furthermore, for high-enough machine penetration, every system exhibits strong emergent security.

Theorem III.1

For any $n\geq 2$, $m\geq 1$, $r\in[0,n]$, $(r^{k})_{k=1}^{m}\in r\Delta(m)$ and any $\alpha\geq 0$, let $\bar{x}(\alpha)$ and $\bar{x}’(\alpha)$ be team equilibria with machine penetration levels $r$ and $r’$, where $r>r’$. Then it holds that

[TABLE]

Furthermore, for every collection of delay functions $\{\tau_{i}\}$, there exists a machine penetration threshold $\bar{r}<n$ such that if $r\geq\bar{r}$, then the system exhibits strong emergent security. That is, for all $\alpha\geq 0$ the resulting team equilibrium (denoted $\bar{x}(r,\alpha)$) is a globally optimal response:

[TABLE]

The proof of Theorem III.1 appears in the Appendix.

That is, Theorem III.1 provides three things: first, when all schedulers have access to all servers, an implication of (7) is that heterogeneous human-machine teams are at least as performant as self-scheduled jobs. Second, for a fixed $\alpha$, increasing the penetration level of machine schedulers can never harm the performance of the system. Third, regardless of the number of machine schedulers, team scheduling is globally optimal provided that the fraction of self-scheduled jobs is small enough. Crucially here, every system may have a positive mass of self-scheduled jobs and still exhibit optimal team scheduling for all attack levels $\alpha$.

It is important to note that the results of Theorem III.1 do not rely in any way on explicit coordination or communication between the various machine schedulers. The only coordination between machine schedulers happens implicitly through the machines’ common objective function.

III-B Closed-form expressions for linear delay functions

Theorem III.1 paints a broad picture of the benefits of machine scheduling (even if uncoordinated), but gives little hint as to the actual performance gains possible. In particular, the only information given about the optimal machine penetration threshold $\bar{r}$ is that $\bar{r}<n$; but the question remains: how much less than $n$?

Accordingly, our next theorem reports the effects of team scheduling in closed-form for a simplified setting. Here, we consider the case that all servers have identical linear delay functions, or $\tau_{i}(x_{i})=x_{i}$. In this setting, it is simple to observe that the nominal optimal scheduling policy under no attack has $x_{i}=1$ for all $i$; since all delay functions are equal, the policy which equates the delay on every server is optimal. The following theorem characterizes the effects of $n$, $r$ and $\alpha$ on the efficiency of team equilibria.

Theorem III.2

For this system, given $\alpha$, it holds that

[TABLE]

When $r\geq\bar{r}$, the system exhibits strong emergent security and we have that every team equilibrium $\bar{x}$ is globally optimal, satisfying

[TABLE]

When $r\in\left(\bar{r}-\frac{\alpha(n-1)}{2n},\bar{r}\right)$, we have that every associated team equilibrium $\bar{x}$ strictly outperforms the fully-selfish equilibrium and satisfies

[TABLE]

Finally, when $r\leq 1-\frac{\alpha(n-1)}{n}$, we have that every associated team equilibrium $\bar{x}$ is equal to a fully selfish equilibrium and satisfies

[TABLE]

That is, for all $r\in[0,n]$, this system exhibits weak emergent security.

The proof of Theorem III.2 appears in the Appendix.

Several things are of note here. First, the penetration threshold $\bar{r}$ is quite low: at most, only a $1/n$ fraction of jobs need to be under the control of machine schedulers to ensure that all team equilibria are optimal. Second, no matter how many jobs are sent to machine schedulers, every team equilibrium is guaranteed to have a relative cost less than $n/(n-1)\leq 2$, regardless of the strength of the attack. Compared to a baseline policy which is unresponsive to attacks, this represents a dramatic potential reduction in cost, since the unresponsive policy has a cost of $1+\alpha/n$, unbounded in $\alpha$. Finally, equation (11) implies that if the machine scheduler penetration is too low, the selfish schedulers entirely dominate the scheduling profile and the machine schedulers have no effect. In fact, for moderate attacks when $\alpha\leq n/(n-1)$, (12) indicates that this situation is identical in cost to the unresponsive baseline scheduling policy $x^{\rm b}$ which simply ignores attacks. Figure 2 depicts the dependence of the normalized system cost on the machine penetration level $r$ for various attack levels $\alpha$.

III-C Loss of security due to scheduling constraints

The result of Theorem III.1 is appealing in its simplicity: if all schedulers (both machine and selfish) have access to scheduling on all servers, then the optimal policy for each machine is simply to behave as though it were the only scheduler and schedule optimally given the choices of others. In this section, we ask if the result would change if some servers were inaccessible to some of the schedulers; we show that even very simple scheduling constraints can render Theorem III.1’s optimal scheduling policies almost entirely ineffective.

Consider the setting of Theorem III.2, in which there are $n\geq 3$ identical servers with delay functions $\tau_{i}(x_{i})=x_{i}$, but now suppose that the self-scheduled traffic can only access servers $1$ and $2$, and that each of the machine schedulers can access any server numbered $2$ through $n$ (that is, machine schedulers can access any server other than $1$). As before, let $r$ denote the mass of jobs controlled by the machine schedulers; to ensure that this system’s optimal operating point is identical to the system of Theorem III.2 for each $\alpha\geq 0$, we enforce an additional constraint that $r=n-1$. See Figure 3 for a depiction of this constrained system.

In this constrained context, our first question is this: if the machine schedulers apply the naive optimal scheduling policy informed by Theorem III.1, how does this system’s security compare to the unconstrained system of Theorem III.2? Theorem III.3 demonstrates that despite the fact that the machine schedulers control the vast majority of jobs, the new constraint completely nullifies the previous benefits of machine scheduling.

Theorem III.3

For $n\geq 3$, in the above system, let an attacker attack server $1$ with attack strength $\alpha\geq 0$. Let $\bar{x}(\alpha)$ be a team equilibrium, and let $x^{\rm n}(\alpha)$ be an uncoordinated self-scheduled equilibrium obtained by converting all jobs to selfish jobs. Then we have

[TABLE]

The proof of Theorem III.3 appears in the Appendix.

That is, in the above system, even if the machine schedulers control a majority fraction of $(n-1)/n$ of the total traffic, simply being unable to access a single machine can dramatically increase the attacker’s ability to harm overall system performance. In fact, for attacks with $\alpha\leq n/(n-1)$, it holds that no team equilibrium outperforms an unresponsive scheduling policy, or $\mathcal{C}^{\alpha}(\bar{x}(\alpha))=\mathcal{C}^{\alpha}(x^{\rm b})$, just as in (12).

To compare (13) with system optimal, consider Figure 4; there, the solid (blue) trace corresponds to the constrained team scheduling in (13), and the dashed (green) trace corresponds to system optimal scheduling.

Note that Theorem III.3 holds for the case that the machine schedulers behave in an uncoordinated manner, each attempting to optimize global performance given the choices of others. What if, on the other hand, a machine scheduler had a “bigger-picture” view of the situation, and could schedule its traffic in anticipation of the self-schedulers’ response?

To understand the effect of such socially-aware machine scheduling, we model this as a Stackelberg equilibrium in which a single machine scheduler selects a policy for servers $\{2,\dots,n\}$ and then the selfish schedulers select between servers $1$ and $2$ on the basis of delay, reminiscent of [21].

Here, we denote the machine-scheduled policy by $x^{\rm m}=(x_{1}^{\rm m},\dots,x_{n}^{\rm m})\in(n-1)\Delta(n)$ and the selfishly-scheduled policy by $x^{\rm s}=(x_{1}^{\rm s},\dots,x_{n}^{\rm s})\in\Delta(n)$, and to model constrained scheduling, we enforce that $x_{1}^{\rm m}=x_{3}^{\rm s}=\cdots=x_{n}^{\rm s}=0$. For each $i$, we let $x_{i}=x_{i}^{\rm m}+x_{i}^{\rm s}$.

The machine schedulers select a policy to minimize the cost function

[TABLE]

Given the machine’s choice of $x^{\rm m}$, the selfish schedulers choose between servers $1$ and $2$ to satisfy the Nash condition (4) given in this case by

[TABLE]

Given $\alpha\geq 0$, we say that a scheduling profile $\left(\hat{x}^{\rm m}(\alpha),\hat{x}^{\rm s}(\alpha)\right)$ is a Stackelberg equilibrium if it satisfies

[TABLE]

where $x^{\rm s}(x^{\rm m})$ denotes that the selfish jobs have scheduled according to (15), given the choice of $x^{\rm m}$.

The following theorem reports the optimal Stackelberg strategy, and demonstrates that only slight security gains are possible in this situation.

Theorem III.4

For $n\geq 3$, in the above system, let an attacker attack server $1$ with attack strength $\alpha\geq 0$. The following scheduling policy enforces an optimal Stackelberg equilibrium satisfying (16):

[TABLE]

and for all $i>2$, $x_{i}^{\rm m}=(n-1-x_{2}^{\rm m})/(n-2).$ Let $\hat{x}(\alpha)$ be an optimal Stackelberg equilibrium enforced by policy (17). Then we have

[TABLE]

The proof of Theorem III.4 appears in the Appendix.

The optimal scheduling policy shown in (17) exhibits the following threshold behavior: when $\alpha$ is below the threshold, the optimal policy is to artificially increase the congestion on server $2$ in an effort to limit access by the selfish jobs. When $\alpha$ increases above the threshold, the optimal policy discontinuously decreases the jobs scheduled on server $2$, since at that point the optimal Stackelberg equilibrium simply equals the team equilibrium from Theorem III.3. That is, when $\alpha$ is above the threshold, the costs of influencing behavior outweigh the benefits, and it becomes optimal to simply allow the self-routers to abandon the attacked server.

Clearly, the normalized cost in the Stackelberg case (18) is always less than the nominal uninfluenced cost from Theorem III.3, but Figure 4 clearly illustrates that this improvement is quite small when compared to the optimal attainable value from (9). That is, Theorem III.4 suggests that the inefficiency introduced by scheduling constraints presents a fundamental impediment to system performance, and is thus deserving of further study.

Furthermore, note that the optimal Stackelberg strategy can optimize for system performance only by deliberately degrading the performance of Server $2$. This is justifiable from the standpoint of overall system performance, since the inefficiencies caused by selfish scheduling are a direct result of too many selfish jobs selecting server $2$. However, the effect of this is that the selfish jobs experience a significantly-degraded quality of service for a wide range of $\alpha$. This can be seen directly in the middle subplot of Figure 5; here, the dash-dot (orange) and solid (blue) traces capture both the load on Server $2$ and the total cost experienced by selfish jobs in Stackelberg an uninfluenced settings, respectively.

IV Conclusion

The results in this paper represent an initial summary look at several possible issues that can arise when distributed autonomous decisionmaking architectures are integrated with those of self-interested human agents. While certain systems are highly receptive to mixed human-autonomous decisionmaking (such as the unconstrained system of Theorems III.1 and III.2), in other types of system, machine scheduling is essentially useless (in the sense that selfish scheduling would perform just as well, as in Theorem III.3). In such systems, even socially-aware machine schedulers (which make decisions in light of their impacts on other decisionmakers) need not improve the situation much. As discussed following Theorem III.4, optimal socially-aware scheduling can even (perhaps paradoxically) cause significant increases in the costs experienced by the human system users. Put together, this paper paints an optimistic picture of the possible benefits of human-machine teams in certain circumstances – but it also tells a cautionary tale of some potential challenges and pitfalls of the teamed paradigm.

Proof of Theorem III.1

The proof hinges on the fact that the problem of finding a team equilibrium can be reduced to finding a Nash flow for a heterogeneous price-sensitive user population in a parallel-network routing game under the influence of marginal-cost tolls. Accordingly, we use techniques and appeal to several results from [22] which pertain to this problem. Given an instance of the team equilibrium problem, construct an instance of the Nash flow routing problem in the following way. Model each server as a network link with delay function $\tau_{i}$. Create two classes of users; one (corresponding to the selfishly scheduled jobs) with mass $n-r$ and cost function on server $i$ simply as $\tau_{i}(x_{i})$. The second class (corresponding to the machine-scheduled jobs) has mass $r$ and cost function on server $i$ given by $\tau_{i}(x_{i})+x_{i}\tau’_{i}(x_{i})$, where $\tau’_{i}$ denotes the first derivative of $\tau_{i}$. That is, the machine-scheduled jobs are modeled by unit-price-sensitive users in a transportation network being charged marginal-cost tolls.

To see that the models are equivalent, simply note that given a choice of server by the selfish jobs, the optimization objective the machine schedulers can be modeled as the potential function of a potential game with server cost functions of $\tau_{i}(x_{i})+x_{i}\tau’_{i}(x_{i})$. That is, if $\bar{x}$ is a team equilibrium, the machine-scheduled jobs can be modeled as nonatomic agents playing a game with cost functions $\tau_{i}(x_{i})+x_{i}\tau’_{i}(x_{i})$. Thus, any team equilibrium of the scheduling problem is a Nash flow for the corresponding routing game.

Given this, we appeal directly to Lemma 2.1 of [22] which states (in the language of the present paper) that for any machine penetration level $r$ and attack level $\alpha\geq 0$, any associated team equilibrium $xt(\alpha)$ satisfies

[TABLE]

proving the first inequality in (7).

To see that (8) is true, note the following. Let $x^{*}$ be an optimal scheduling profile when $\alpha=0$. Let $k$ be the index of a minimum-delay server under profile $x^{*}$, or

[TABLE]

Observe that at this optimal profile, if all jobs currently scheduled on server $k$ are “converted” to self-scheduled jobs, they will still (at least weakly) prefer server $k$, since by definition, for all $j\geq 1$,

[TABLE]

That is, setting $\bar{r}=\sum_{i\neq k}x_{i}^{*}$, there exists an optimal team equilibrium $\bar{x}$ with $r=\bar{r}<n$ since the $n-\bar{r}$ selfish jobs can be considered to be selecting server $k$. Note that even if $k=1$, it must be true that $x^{*}_{1}\leq\bar{r}$. For the remainder of the proof, any team equilibrium is assumed to be for a population in which $r=\bar{r}$.

As discussed in the proof of Lemma 5.2 in [22], for this population it holds that for any nontrivial attack with $\alpha>0$ that in any team equilibrium, selfish schedulers will select server $1$ (the attacked server) if and only if no machine schedulers select any other server, or $\bar{x}_{1}(\alpha)>\bar{r}$. However, since all delay functions (and their marginal costs) are nondecreasing, any team equilibrium with $\alpha>0$ must have $\bar{x}_{1}(\alpha)<\bar{x}_{1}(0)$ and $\bar{x}_{i}(\alpha)\geq\bar{x}_{i}(0)$ for all $i>1$. Due to these monotonicity properties, since $\bar{x}(0)=x^{*}$, we thus have that for all $\alpha>0$, $\bar{x}_{1}(\alpha)\leq\bar{r}$, meaning that no selfish schedulers select server $1$ in any team equilibrium.

To complete the proof of (8), we show that when $\alpha>0$, in every $\bar{x}(\alpha)$ it must be the case that every server has a positive mass of machine-scheduled jobs. This is because the mass of self-scheduled jobs is $n-\min_{i}\{x_{i}(0)\}$, and for all $i>1$, $\alpha>0$ implies that $\bar{x}_{i}(\alpha)>\bar{x}_{i}(0)$. Recalling again the previously-referenced statements from [22], if all servers have a positive mass of machine-scheduled jobs, the team equilibrium must satisfy (8). ∎

Proof of Theorem III.2

First we show that for any machine penetration level $r$ and attack level $\alpha$, every associated team equilibrium $\bar{x}(\alpha)$ has

[TABLE]

Suppose that $x$ is an aggregate scheduling profile in which $x_{i}>x_{j}$ for some $i,j>1$. Note that if any selfish jobs are scheduled on server $i$, then (4) implies that $x$ is not a team equilibrium, since there exists a server with lower cost (namely, server $j$): $\tau_{j}(x_{j})<\tau_{i}(x_{i})$. Further, note that $\tau_{j}(x_{j})<\tau_{i}(x_{i})$ does not optimize the machine objective of (3) since a small mass of jobs transferred from $i$ to $j$ would decrease the system’s total cost. That is, if any machine has scheduled on server $i$, it follows that $x$ is not a team equilibrium. Thus, for $i,j>1$, every team equilibrium has $\bar{x}_{i}=\bar{x}_{j}$.

For any $\alpha\geq 0$, it is straightforward to verify that the globally optimal scheduling policy $x^{*}$ has $x^{*}_{1}=\max\{0,1-\frac{\alpha(n-1)}{2n}\}$, and the remaining jobs divided evenly among servers $2$ through $n$, so that for $i>1$, $x_{i}^{*}=\min\{n/(n-1),1+\frac{\alpha}{2n}\}$. Let the machine penetration level meet or exceed $x_{1}^{*}$: $r\geq 1-\frac{\alpha(n-1)}{2n}$. Given this $r$, let $x$ be an aggregate profile with $x_{1}<x_{1}^{*}$ and the remaining jobs divided equally among servers $2$ through $n$. In $x$, some machine must be scheduling a positive mass of jobs on a server other than $1$; this machine could improve the value of its objective by shifting some jobs to server $1$ to bring $x_{1}$ closer to $x_{1}^{*}$ – implying that $x$ is not a team equilibrium. On the other hand, let $x^{\prime}$ be an aggregate profile with $x^{\prime}_{1}>x_{1}^{*}$ and the remaining jobs divided equally among servers $2$ through $n$. In $x^{\prime}$, it is straightforward to verify that $\tau_{1}(x^{\prime}_{1})+\alpha>\tau_{2}(x^{\prime}_{2})$, implying that if selfish jobs are selecting $1$, $x^{\prime}$ cannot be a team equilibrium. Further, if all of the jobs on server $1$ are machine-scheduled, some of them could be shifted to servers $2$ through $n$ to bring the total cost of the profile closer to optimal. Thus, $x^{\prime}$ is not a team equilibrium, and it must be the case that when $r\geq 1-\frac{\alpha(n-1)}{2n}$, every team equilibrium is globally-optimal; (9) may be verified by substituting the aforementioned values of $x_{1}^{*}$ and $x_{i}^{*}$ into (2).

The proofs of the remaining statements in Theorem III.2 follow similar logic. For (10), there are not enough machine schedulers to ensure that $\bar{x}_{1}=x_{1}^{*}$, but there are enough to ensure that no selfish jobs schedule on server $1$. Thus, in this intermediate case, every team equilibrium has $\bar{x}_{1}\in\{0,r\}$ and $\bar{x}_{i}=(n-\bar{x}_{1})/(n-1)$, and (10) follows algebraically.

Finally, for (11), there are so few machine-scheduled jobs in this case that the selfish jobs are all indifferent between all links and ensure that $\tau_{1}(\bar{x}_{1})+\alpha=\tau_{i}(\bar{x}_{i})$ for all $i\geq 1$, yielding $\bar{x}_{1}=\max\{1-\alpha(n-1)/n,0\}$ and (11). ∎

Proof of Theorem III.3

Here, we may repeat the argument from the proof of Theorem III.2 that any team equilibrium in this case has

[TABLE]

The key feature of Theorem III.2 that allowed a small mass of machine-scheduled jobs to enforce optimality was that the machine-scheduled jobs could access the attacked server and effectively displace the selfish jobs. In the present constrained setting, this is no longer possible since machine schedulers cannot access server $1$.

Thus, the equilibrium condition of the selfish jobs guarantees that for low $\alpha$, in any team equilibrium $\bar{x}(\alpha)$, $\bar{x}_{1}+\alpha=\bar{x}_{2}$. Combining this with (22) ensures that $\bar{x}(\alpha)$ is identical to a fully-selfish equilibrium $x^{\rm n}(\alpha)$, and (13) may be computed as in the third part of the proof of Theorem III.2. ∎

Proof of Theorem III.4

Lemma .1

For every $\alpha\geq 0$, there exists an optimal Stackelberg equilibrium in which

[TABLE]

Proof:

This is immediately true for $\alpha=0$, when the unique optimal Stackelberg equilibrium has $\hat{x}_{i}=1$ for all $i$. Therefore, let $\alpha>0$ and let $x$ be any scheduling profile such that $x_{1}+\alpha<x_{2}$. If $x$ is a Stackelberg equilibrium, the selfish schedulers’ incentive constraint in (16) implies that ${x}_{2}=0$, or that ${x}_{1}=1$. From here, it is simple to show that the normalized cost of this is lower-bounded by

[TABLE]

satisfied with equality when $x_{i}=1$ for all $i=1$.

We will show a machine-scheduled policy $\tilde{x}^{\rm m}$ which weakly outperforms profile $x$ and always has $\tilde{x}^{\rm m}_{1}+\alpha\geq\tilde{x}^{\rm m}_{2}$, implying that the optimal Stackelberg equilibrium can always be selected to satisfy $x_{1}+\alpha\geq x_{2}$. Let $\tilde{x}^{\rm m}_{2}=1/(n-1)$, and let $\tilde{x}^{\rm m}_{i}=n/(n-1)$ for all $i>2$. Clearly, for all $\alpha$, $\tilde{x}_{1}^{\rm s}=1$ cannot constitute a Nash response by the self-scheduled jobs, as this would yield $\tilde{x}_{1}+\alpha>\tilde{x}_{2}$, and selfish jobs would have a strong incentive to deviate to server $2$. Thus, it must be the case that $\tilde{x}_{2}^{\rm s}>0$, which by the Nash condition in (15) implies that $\tilde{x}_{1}+\alpha\geq\tilde{x}_{2}$. Further manipulations yield the fact that $\mathcal{C}^{\alpha}(\tilde{x})\leq 1+{\alpha}/{n}.$ Clearly, any optimal Stackelberg equilibrium $\hat{x}$ must weakly outperform $\tilde{x}$, implying that for every $\alpha\geq 0$, there exists an optimal Stackelberg equilibrium $\hat{x}$ satisfying (23). ∎

Proof of Theorem III.4.

An optimal Stackelberg equilibrium $\hat{x}$ is a solution of the following optimization problem:

[TABLE]

The first constraint represents flow conservation; the second constraint is a consequence of Lemma .1. It can easily be shown that the optimum must have $x_{i}>0$ for all $i>1$, leaving only the relevant nonnegativity constraint of $x_{1}\geq 0$. Associating KKT multipliers $\lambda\geq 0$, $\mu_{1}\geq 0$ and $\mu_{2}\geq 0$ with the three constraints in (25), we have first-order optimality conditions given by

[TABLE]

Note the following:

$\mu_{1}>0$ implies that $x_{1}+\alpha=x_{2}$, which (due to the selfish jobs’ Nash conditions) implies that $x_{1}\geq 0$, so we may consider $\mu_{2}=0$. 2. 2.

$x_{1}+\alpha>x_{2}$ implies that $x_{1}=0$, in which case $\mu_{2}\geq 0$.

That is, without loss of generality we may assume that $\mu_{1}>0\iff\mu_{2}=0$. With that in hand, when $\mu_{1}>0$ (and $\mu_{2}=0$), the local optimizer $x^{*}(\alpha)$ (for $\alpha\leq 4n/(3n-2)$) satisfies

[TABLE]

and the corresponding normalized cost can be derived as

[TABLE]

When $\mu_{1}=0$ and $\mu_{2}>0$, the resulting equilibrium resolves to the under-influenced team equilibrium $\bar{x}(\alpha)$ of Theorem III.3, with normalized cost satisfying that of (13).

Thus, the optimal Stackelberg equilibrium $\hat{x}(\alpha)$ has normalized cost equal to that in (18). Finally, equations (26)-(28), combined with the selfish jobs’ conservation constraint, can be manipulated to show (17). ∎