Strategic Learning for Active, Adaptive, and Autonomous Cyber Defense

TL;DR

This paper proposes a new active, autonomous, and adaptive cyber defense framework called '3A', utilizing strategic learning schemes to improve defense effectiveness under uncertain information conditions.

Contribution

It introduces three defense schemes with varying information restrictions and applies strategic learning to optimize their policies in uncertain environments.

Findings

All three schemes converge to optimal policies through reinforcement.

The framework effectively balances security and operational costs.

It provides a foundation for proactive, strategic cyber defense under incomplete information.

Abstract

The increasing instances of advanced attacks call for a new defense paradigm that is active, autonomous, and adaptive, named as the \texttt{`3A'} defense paradigm. This chapter introduces three defense schemes that actively interact with attackers to increase the attack cost and gather threat information, i.e., defensive deception for detection and counter-deception, feedback-driven Moving Target Defense (MTD), and adaptive honeypot engagement. Due to the cyber deception, external noise, and the absent knowledge of the other players' behaviors and goals, these schemes possess three progressive levels of information restrictions, i.e., from the parameter uncertainty, the payoff uncertainty, to the environmental uncertainty. To estimate the unknown and reduce uncertainty, we adopt three different strategic learning schemes that fit the associated information restrictions. All three learning schemes share the same feedback structure of sensation, estimation, and actions so that the most rewarding policies get reinforced and converge to the optimal ones in autonomous and adaptive fashions. This work aims to shed lights on proactive defense strategies, lay a solid foundation for strategic learning under incomplete information, and quantify the tradeoff between the security and costs.

Equations

$$b_i^{k+1}({\theta }_j|h^k\cup \{a_i^k,a_j^k\},{\theta }_i)=\frac{{\sigma }_i^k(a_i^k|h^k,{\theta }_i){\sigma }_j^k(a_j^k|h^k,{\theta }_j)b_i^k({\theta }_j|h^k,{\theta }_i)}{{\sum }_{{\bar{\theta }}_j\in {\Theta }_j}{\sigma }_i^k(a_i^k|h^k,{\theta }_i){\sigma }_j^k(a_j^k|h^k,{\bar{\theta }}_j)b_i^k({\bar{\theta }}_j|h^k,{\theta }_i)}.$$

$$b_i^{k+1}({\theta }_j|x^{k+1},{\theta }_i)=\frac{\mathrm{Pr}(x^{k+1}|{\theta }_j,x^k,{\theta }_i)b_i^k({\theta }_j|x^k,{\theta }_i)}{{\sum }_{{\bar{\theta }}_j\in {\Theta }_j}\mathrm{Pr}(x^{k+1}|{\bar{\theta }}_j,x^k,{\theta }_i)b_i^k({\bar{\theta }}_j|x^k,{\theta }_i)}.$$

$$U_i^{k_0:K}({\sigma }_i^{k_0:K},{\sigma }_j^{k_0:K},x^{k_0},{\theta }_i):=\sum _{k=k_0}^K{E}_{{\theta }_j\sim b_i^k,a_i^k\sim {\sigma }_i^k,a_j^k\sim {\sigma }_j^k}{J}_i^k(x^k,a_1^k,a_2^k,{\theta }_1,{\theta }_2).$$

$$\begin{array}{rl}& U_1^{k_0:K}({\sigma }_1^{\ast ,k_0:K},{\sigma }_2^{\ast ,k_0:K},x^{k_0},{\theta }_1)+\epsilon \ge U_1^{k:K}({\sigma }_1^{k_0:K},{\sigma }_2^{\ast ,k_0:K},x^{k_0},{\theta }_1),\\ & U_2^{k_0:K}({\sigma }_1^{\ast ,k_0:K},{\sigma }_2^{\ast ,k_0:K},x^{k_0},{\theta }_2)+\epsilon \ge U_2^{k:K}({\sigma }_1^{\ast ,k_0:K},{\sigma }_2^{k_0:K},x^{k_0},{\theta }_2).\end{array}$$

$$r_l(a_{l,j},c_{l,i})=\{\begin{array}{ll}{D}_{ij},& {\gamma }_l^{-1}(a_{l,j})\in {\pi }_l(c_{l,k})\\ 0,& \text{otherwise}\end{array}.$$

$$\text{mathbbm}{r}_l({\mathbf{f}}_l^{\ast },{\mathbf{g}}_l)\le \text{mathbbm}{r}_l({\mathbf{f}}_l^{\ast },{\mathbf{g}}_l^{\ast })\le \text{mathbbm}{r}_l({\mathbf{f}}_l,{\mathbf{g}}_l^{\ast }),\forall {\mathbf{f}}_l\in △{\mathcal{C}}_l,{\mathbf{g}}_l\in △{\mathcal{A}}_l,$$

$$\text{mathbbm}{r}_l({\mathbf{f}}_l,{\mathbf{g}}_l):={\mathbb{E}}_{{\mathbf{f}}_l,{\mathbf{g}}_l}{r}_l=\sum _{k=1}^{n_l}\sum _{h=1}^{m_l}{f}_{l,h}{g}_{l,k}{r}_l(a_{l,k},c_{l,h}).$$

$$\begin{array}{rl}& {\hat{r}}_{l,t+1}^S(c_{l,h})={\hat{r}}_{l,t}^S(c_{l,h})+{\mu }_t^S{1}_{\{\text{mathbbm}{c}_{l,t}=c_{l,h}\}}(r_{l,t}-{\hat{r}}_{l,t}^S(c_{l,h})),\\ & {\hat{r}}_{l,t+1}^A(a_{l,h})={\hat{r}}_{l,t}^A(a_{l,h})+{\mu }_t^A{1}_{\{\text{mathbbm}{a}_{l,t}=a_{l,h}\}}(r_{l,t}-{\hat{r}}_{l,t}^A(a_{l,h})).\end{array}$$

$$R_{l,t}^S:=\sum _{h=1}^{m_l}{f}_{l,h,t+1}\ln \left(\frac{f_{l,h,t+1}}{f_{l,h,t}}\right).$$

$$(\text{SP}):\underset{{\mathbf{f}}_{l,t+1}\in △{\mathcal{C}}_l}{\sup }-\sum _{h=1}^{m_l}{f}_{l,h,t+1}{\hat{r}}_{l,t}^S(c_{l,h})-{ϵ}_{l,t}^S{R}_{l,t}^S.$$

$$(\text{AP}):\underset{{\mathbf{g}}_{l,t+1}\in △{\mathcal{A}}_l}{\sup }-\sum _{h=1}^{n_l}{g}_{l,h,t+1}{\hat{r}}_{l,t}^A(a_{l,h})-{ϵ}_{l,t}^A\sum _{h=1}^{n_l}{g}_{l,h,t+1}\ln \left(\frac{g_{l,h,t+1}}{g_{l,h,t}}\right).$$

$$\begin{array}{rl}& f_{l,h,t+1}=\frac{f_{l,h,t}{e}^{-\frac{{\hat{r}}_{l,t}(c_{l,h})}{{ϵ}_{l,t}^S}}}{\sum _{h^{\prime }=1}^{m_l}{f}_{l,h^{\prime },t}{e}^{-\frac{{\hat{r}}_{l,t}(c_{l,h^{\prime }})}{{ϵ}_{l,t}^S}}},g_{l,h,t+1}=\frac{g_{l,h,t}{e}^{-\frac{{\hat{r}}_{l,t}(a_{l,h})}{{ϵ}_{l,t}^A}}}{\sum _{h^{\prime }=1}^{n_l}{g}_{l,h^{\prime },t}{e}^{-\frac{{\hat{r}}_{l,t}(a_{l,h^{\prime }})}{{ϵ}_{l,t}^A}}},\\ & W_{l,t}^S={ϵ}_{l,t}^S\ln \left(\sum _{h=1}^{m_l}{f}_{l,h,t}{e}^{-\frac{{\hat{r}}_{l,t}(c_{l,h})}{{ϵ}_{l,t}^S}}\right),W_{l,t}^A={ϵ}_{l,t}^A\ln \left(\sum _{h=1}^{n_l}{g}_{l,h,t}{e}^{-\frac{{\hat{r}}_{l,t}(a_{l,h})}{{ϵ}_{l,t}^A}}\right).\end{array}$$

$$\begin{array}{r}{f}_{l,h,t+1}=(1-{\lambda }_{l,t}^S)f_{l,h,t}+{\lambda }_{l,t}^S\frac{f_{l,h,t}{e}^{-\frac{{\hat{r}}_{l,t}(c_{l,h})}{{ϵ}_{l,t}^S}}}{\sum _{h^{\prime }=1}^{m_l}{f}_{l,h^{\prime },t}{e}^{-\frac{{\hat{r}}_{l,t}(c_{l,h^{\prime }})}{{ϵ}_{l,t}^S}}},\\ g_{l,h,t+1}=(1-{\lambda }_{l,t}^A)g_{l,h,t}+{\lambda }_{l,t}^A\frac{g_{l,h,t}{e}^{-\frac{{\hat{r}}_{l,t}(a_{l,h})}{{ϵ}_{l,t}^A}}}{\sum _{h^{\prime }=1}^{n_l}{g}_{l,h^{\prime },t}{e}^{-\frac{{\hat{r}}_{l,t}(a_{l,h^{\prime }})}{{ϵ}_{l,t}^A}}}.\end{array}$$

$$\sum _{t=1}^{\infty }{x}_t=+\infty ,\sum _{t=1}^{\infty }(x_t{)}^2<+\infty .$$

$$\begin{array}{rl}& \frac{d}{dt}{f}_{l,h,t}=f_{l,h,t}\left(\frac{e^{-\frac{{\hat{r}}_{l,t}(c_{l,h})}{{ϵ}_{l,t}^S}}}{\sum _{h^{\prime }=1}^{m_l}{f}_{l,h^{\prime },t}{e}^{-\frac{{\hat{r}}_{l,t}(c_{l,h^{\prime }})}{{ϵ}_{l,t}^S}}}-1\right),h=1,2,\cdots ,m_l,\\ & \frac{d}{dt}{\hat{r}}_{l,t}^S(c_{l,h})=-\text{mathbbm}{r}_{l,t}(e_{c_{l,h}},{\mathbf{g}}_{l,t})-{\hat{r}}_{l,t+1}^S(c_{l,h}),c_{l,h}\in {\mathcal{C}}_l.\end{array}$$

$$\begin{array}{rl}& \frac{d}{dt}{g}_{l,h,t+1}=g_{l,h,t}\left(\frac{e^{-\frac{{\hat{r}}_{l,t}(a_{l,h})}{{ϵ}_{l,t}^A}}}{\sum _{h^{\prime }=1}^{n_l}{g}_{l,h^{\prime },t}{e}^{-\frac{{\hat{r}}_{l,t}(a_{l,h^{\prime }})}{{ϵ}_{l,t}^A}}}-1\right),h=1,2,\cdots ,n_l,\\ & \frac{d}{dt}{\hat{r}}_{l,t+1}^A(a_{l,h})=\text{mathbbm}{r}_{l,t}({\mathbf{f}}_{l,t},e_{a_{l,h}})-{\hat{r}}_{l,t+1}^A(a_{l,h}),a_{l,h}\in {\mathcal{A}}_l.\end{array}$$

$$u(s^0,\pi )=E[\sum _{k=0}^{\infty }{\int }_{T^k}^{T^{k+1}}{e}^{-\gamma (\tau +T^k)}(r(S^k,A^k,S^{k+1},T^k,T^{k+1},\tau )+w_r)d\tau ].$$

$$v(s^0)=\underset{a^0\in \mathcal{A}(s^0)}{\sup }E[{\int }_{T^0}^{T^1}{e}^{-\gamma (\tau +T^0)}r(s^0,a^0,S^1,T^0,T^1,\tau )d\tau +e^{-\gamma T^1}v(S^1)].$$

$$v(s^0)=\underset{a^0\in \mathcal{A}(s^0)}{\sup }\sum _{s^1\in \mathcal{S}}tr(s^1|s^0,a^0)(r^{\gamma }(s^0,a^0,s^1)+z^{\gamma }(s^0,a^0,s^1)v(s^1)),$$

$$\sum _{s^1\in \mathcal{S}}tr(s^1|s^0,a^0)z(\delta |s^0,a^0,s^1)\le 1-\theta ,\forall s^0\in \mathcal{S},a^0\in \mathcal{A}(s^0).$$

$$\begin{array}{rl}{Q}^{k+1}({\bar{s}}^k,a^k)=& (1-{\alpha }^k({\bar{s}}^k,a^k))Q^k({\bar{s}}^k,a^k)+{\alpha }^k({\bar{s}}^k,a^k)[{\bar{r}}_1({\bar{s}}^k,a^k,{\bar{s}}^{k+1})\\ & +{\bar{r}}_2({\bar{s}}^k,a^k)\frac{(1-e^{-\gamma {\bar{\tau }}^k})}{\gamma }-e^{-\gamma {\bar{\tau }}^k}\underset{a^{\prime }\in \mathcal{A}({\bar{s}}^{k+1})}{\max }{Q}^k({\bar{s}}^{k+1},a^{\prime })],\end{array}$$

Full text

1 Introduction

Recent instances of WannaCry ransomware, Petya cyberattack, and Stuxnet malware have demonstrated the trends of modern attacks and the corresponding new security challenges as follows.

• •

Advanced: Attackers leverage sophisticated attack tools to invalidate the off-the-shelf defense schemes such as the firewall and intrusion detection systems.

• •

Targeted: Unlike automated probes, targeted attacks conduct thorough research in advance to expose the system architecture, valuable assets, and defense schemes.

• •

Persistent: Attackers can restrain the adversary’s behaviors and bide their times to launch critical attacks. They are persistent in achieving the goal.

• •

Adaptive: Attackers can learn the defense strategies and unpatched vulnerabilities during the interaction with the defender and tailor their strategies accordingly.

• •

Stealthy and Deceptive: Attackers conceal their true intentions and disguise their claws to evade detection. The adversarial cyber deception endows attackers an information advantage over the defender.

Thus, defenders are urged to adopt active, adaptive, and autonomous defense paradigms to deal with the above challenges and proactively protect the system prior to the attack damages rather than passively compensate for the loss. In analogy to the classical Kerckhoffs’s principle in the 19th century that attackers know the system, we suggest a new security principle for modern cyber systems as follows: {svgraybox} Principle of 3A Defense: A cyber defense paradigm is considered to be insufficiently secure if its effectiveness relies on

• •

Rule-abiding human behaviors.

• •

A perfect protection against vulnerabilities and a perfect prevention from system penetration.

• •

A perfect knowledge of attacks.

Firstly, $30\%$ of data breaches are caused by privilege misuse and error by insiders according to Verizon’s data breach report in $2019$ Jeff2000 . Security administration does not work well without the support of technology, and autonomous defense strategies are required to deal with the increasing volume of sophisticated attacks. Secondly, systems always have undiscovered vulnerabilities or unpatched vulnerabilities due to the long supply chain of uncontrollable equipment providers shackleford2015combatting and the increasing complexities in the system structure and functionality. Thus, an effective paradigm should assume a successful infiltration and pursue strategic securities through interacting with intelligent attackers. Finally, due to adversarial deception techniques and external noises, the defender cannot expect a perfect attack model with predicable behaviors. The defense mechanism should be robust under incomplete information and adaptive to the evolution of attacks.

In this chapter, we illustrate three active defense schemes in our previous works, which are designed based on the new cyber security principle. They are defensive deception for detection and counter-deception huang2019adaptive ; huang2018analysis ; APTjournal in Section 2, feedback-driven Moving Target Defense (MTD) zhu2013game in Section 3, and adaptive honeypot engagement huangHoneypot in Section 4. All three schemes is of incomplete information, and we arrange them based on three progressive levels of information restrictions as shown in the left part of Fig. 1.

The first scheme in Section 2 considers the obfuscation of characteristics of known attacks and systems through a random parameter called the player’s type. The only uncertainty origins from the player’s type, and the mapping from the type to the utility is known deterministically. The MTD scheme in Section 3 considers unknown attacks and systems whose utilities are completely uncertain, while the honeypot engagement scheme in Section 4 further investigates environmental uncertainties such as the transition probability, the sojourn distribution, and the investigation reward.

To deal with these uncertainties caused by different information structures, we suggest three associated learning schemes as shown in the right part of Fig. 1, i.e., Bayesian learning for the parameter estimation, distributed learning for the utility acquisition without information sharing, and reinforcement learning for the optimal policy obtainment under the unknown environment. All three learning methods form a feedback loop that strategically incorporates the samples generated during the interaction between attackers and defenders to persistently update the beliefs of known and then take actions according to current optimal decision strategies. The feedback structure makes the learning adaptive to behavioral and environmental changes.

Another common point of these three schemes is the quantification of the tradeoff between security and the different types of cost. In particular, the costs result from the attacker’s identification of the defensive deception, the system usability, and the risk of attackers penetrating production systems from the honeynet, respectively.

1.1 Literature

The idea of using deceptions defensively to detect and deter attacks has been studied theoretically as listed in the taxonomic survey pawlick2017game , implemented to the Adversarial Tactics, Techniques and Common Knowledge (ATT&CKTM) adversary model system stech2016integrating , and tested in the real-time cyber-wargame experiment heckman2013active . Many previous works imply the similar idea of type obfuscation, e.g., creating social network avatars (fake personas) on the major social networks gomez2018r , implementing honey files for ransomware actions virvilis2014changing , and disguising a production system as a honeypot to scare attackers away pawlick2018modeling .

Moving target defense (MTD) allows dynamic security strategies to limit the exposure of vulnerabilities and the effectiveness of the attacker’s reconnaissance by increasing complexities and costs of attacks jajodia2011moving . To achieve an effective MTD, kc2003countering proposes the instruction set and the address space layout randomization, clark2012deceptive studies the deceptive routing against jamming in multi-hop relay networks, and maleki2016markov uses the Markov chain to model the MTD process and discusses the optimal strategy to balance the defensive benefit and the network service quality.

The previous two methods use the defensive deception to protect the system and assets. To further gather threat information, the defender can implement honeypots to lure attackers to conduct adversarial behaviors and reveal their TTPs in a controlled and monitored environment. Previous works hecker2012methodology ; la2016deceptive have investigated the adaptive honeypot deployment to effectively engage attackers without their notices. The authors in recent work PawlickNZ17 proposes a continuous-state Markov Decision Process (MDP) model and focuses on the optimal timing of the attacker ejection.

Game-theoretic models are natural frameworks to capture the multistage interaction between attackers and defenders. Recently, game theory has been applied to different sets of security problems, e.g., Stackelberg and signaling games for deception and proactive defenses pawlick_stackelberg_2016 ; zhu2013game ; zhu2013deployment ; zhu2013hybrid ; zhu2012interference ; clark2012deceptive ; zhu2012game ; zhu2012deceptive ; zhu2010stochastic , network games for cyber-physical security xu2017secure ; xu_game-theoretic_2017 ; xu_cross-layer_2016 ; farooq2019modeling ; xu2015cyber ; huang2017large ; chen2017dynamic ; miao2018hybrid ; yuan2013resilient ; Rass&Zhu2016 , dynamic games for adaptive defense zhu2010dynamic ; zhang2017strategic ; huang2018gamesec ; huang2018PER ; huang2019adaptive ; pawlick2015flip ; farhang2014dynamic ; zhu2009dynamic ; zhu2010network ; zhu2010heterogeneous , and mechanism design theory for security chen_security_2017 ; zhang_bi-level_2017 ; zhang_attack-aware_2016 ; casey2015compliance ; hayel2015attack ; hayel2017epidemic ; zhu2012guidex ; zhu2012tragedy ; zhu2009game .

Information asymmetry among the players in network security is a challenge to deal with. The information asymmetry can be either leveraged or created by the attacker or the defender for achieving a successful cyber deception. For example, techniques such as honeynets carroll2011game ; zhu2013deployment , moving target defense zhu2013game ; jajodia2011moving ; huang2019adaptive , obfuscation pawlick2016stackelberg ; zhang_dynamic_2017 ; farhang2015phy ; zhang2018distributed , and mix networks zhang2010gpath have been introduced to create difficulties for attackers to map out the system information.

To overcome the created or inherent uncertainties of networks, many works have studied the strategic learning in security games, e.g., Bayesian learning for unknown adversarial strategies garnaev2015security , heterogeneous and hybrid distributed learning zhu2010heterogeneous ; zhu2011distributed , multiagent reinforcement learning for intrusion detection servin2008multi . Moreover, these learning schemes are combined to achieve better properties, e.g., distributed Bayesian learning djuric2012distributed , Bayesian reinforcement learning chalkiadakis2003coordination , and distributed reinforcement learning chen2015distributed .

1.2 Notation

Throughout the chapter, we use calligraphic letter $\mathcal{A}$ to define a set and $|\mathcal{A}|$ as the cardinality of the set. Let $\bigtriangleup\mathcal{A}$ represent the set of probability distributions over $\mathcal{A}$. If set $\mathcal{A}$ is discrete and finite, $\bigtriangleup\mathcal{A}:=\{p:\mathcal{A}\mapsto R_{+}|\sum_{a\in\mathcal{A}}p(a)=1\}$, otherwise, $\bigtriangleup\mathcal{A}:=\{p:\mathcal{A}\mapsto R_{+}|\int_{a\in\mathcal{A}}p(a)=1\}$. Row player $P_{1}$ is the defender (pronoun ‘she’) and $P_{2}$ (pronoun ‘he’) is the user (or the attacker) who controls the column of the game matrix. Both players want to maximize their own utilities. The indicator function $\mathbf{1}_{\{x=y\}}$ equals one if $x=y$, and zero if $x\neq y$.

1.3 Organization of the Chapter

The rest of the paper is organized as follows. In Section 2, we elaborate defensive deception as a countermeasure of the adversarial deception under a multistage setting where Bayesian learning is applied for the parameter uncertainty. Section 3 introduces a multistage MTD framework and the uncertainties of payoffs result in distributed learning schemes. Section 4 further considers reinforcement learning for environmental uncertainties under the honeypot engagement scenario. The conclusion and discussion are presented in Section 5.

2 Bayesian Learning for Uncertain Parameters

Under the mild restrictive information structure, each player’ utility is completely governed by a finite group of parameters which form his/her type. Each player’s type characterizes all the uncertainties about this player during the game interaction, e.g., the physical outcome, the payoff function, and the strategy feasibility, as an equivalent utility uncertainty without loss of generality harsanyi1967games . Thus, the revelation of the type value directly results in a game of complete information. In the cyber security scenario, a discrete type can distinguish either systems with different kinds of vulnerabilities or attackers with different targets. The type can also be a continuous random variable representing either the threat level or the security awareness level huang2019adaptive ; huang2018analysis . Since each player $P_{i}$ takes actions to maximize his/her own type-dependent utility, the other player $P_{j}$ can form a belief to estimate $P_{i}$’s type based on the observation of $P_{i}$’s action history. The utility optimization under the beliefs results in the Perfect Bayesian Nash Equilibrium (PBNE) which generates new action samples and updates the belief via the Bayesian rule. We plot the feedback Bayesian learning process in Fig. 2 and elaborate each element in the following subsections based on our previous work APTjournal .

2.1 Type and Multistage Transition

Through adversarial deception techniques, attackers can disguise their subversive actions as legitimate behaviors so that the defender $P_{1}$ cannot judge whether a user $P_{2}$’s type $\theta_{2}\in\Theta_{2}:=\{\theta_{2}^{g},\theta_{2}^{b}\}$ is legitimate $\theta_{2}^{g}$ or adversarial $\theta_{2}^{b}$. As a countermeasure, the defender can introduce the defensive deception so that the attacker cannot distinguish between a primitive system $\theta_{1}^{L}$ and a sophisticated system $\theta_{1}^{H}$, i.e., the defender has a binary type $\theta_{1}\in\Theta_{1}:=\{\theta_{1}^{H},\theta_{1}^{L}\}$. A sophisticated system is costly yet deters attacks and causes damages to attackers. Thus, a primitive system can disguise as a sophisticated one to draw the same threat level to attackers yet avoid the implementation cost of sophisticated defense techniques.

Many cyber networks contain hierarchical layers, and up-to-date attackers such as Advanced Persistent Threats (APTs) aim to penetrate these layers and reach specific targets at the final stage as shown in Fig. 3.

At stage $k\in\{0,1,\cdots,K\}$, $P_{i}$ takes an action $a_{i}^{k}\in\mathcal{A}_{i}^{k}$ from a finite and discrete set $\mathcal{A}_{i}^{k}$. Both players’ actions become fully observable after applied and each action does not directly reveal the private type. For example, both legitimate and adversarial users can choose to access the sensor, and both primitive and sophisticated defenders can choose to monitor the sensor. Both players’ actions up to stage $k$ constitute the history $h^{k}=\{a_{1}^{0},\cdots,a_{1}^{k-1},a_{2}^{0},\cdots,a_{2}^{k-1}\}\in\mathcal{H}^{k}:=\prod_{i=1}^{2}\prod_{\bar{k}=0}^{k-1}\mathcal{A}_{i}^{\bar{k}}$. Given history $h^{k}$ at the current stage $k$, players at stage $k+1$ obtain an updated history $h^{k+1}=h^{k}\cup\{a_{1}^{k},a_{2}^{k}\}$ after the observation $a_{1}^{k},a_{2}^{k}$. A state $x^{k}\in\mathcal{X}^{k}$ at each stage $k$ is the smallest set of quantities that summarize information about actions in previous stages so that the initial state $x^{0}\in\mathcal{X}^{0}$ and the history at stage $k$ uniquely determine ${x}^{k}$ through a known state transition function $f^{k}$, i.e., $x^{k+1}=f^{k}(x^{k},a_{1}^{k},a_{2}^{k}),\forall k\in\{0,1,\cdots,K-1\}$. The state can represent the location of the user in the attack graph, and also other quantities such as users’ privilege levels and status of sensor failures.

A behavioral strategy $\sigma^{k}_{i}\in\Sigma^{k}_{i}:\mathcal{I}^{k}_{i}\mapsto\bigtriangleup(\mathcal{A}_{i}^{k})$ maps $P_{i}$’s information set $\mathcal{I}^{k}_{i}$ at stage $k$ to a probability distribution over the action space $\mathcal{A}_{i}^{k}$. At the initial stage [math], since the only information available is the player’s type realization, the information set $\mathcal{I}_{i}^{0}=\Theta_{i}$. The action is a realization of the behavioral strategy, or equivalently, a sample drawn from the probability distribution $\sigma^{k}_{i}(\cdot|I_{i}^{k})$. With a slight abuse of notation, we denote $\sigma^{k}_{i}(a_{i}^{k}|{I}_{i}^{k})$ as the probability of $P_{i}$ taking action $a_{i}^{k}\in\mathcal{A}_{i}^{k}$ given the available information $I_{i}^{k}\in\mathcal{I}_{i}^{k}$.

2.2 Bayesian Update under Two Information Structure

Since the other player’s type is of private information, $P_{i}$ forms a belief $b^{k}_{i}:\mathcal{I}_{i}^{k}\mapsto\bigtriangleup(\Theta_{j}),j\neq i$, on $P_{j}$’s type using the available information $\mathcal{I}^{k}_{i}$. Likewise, given information $I_{i}^{k}\in\mathcal{I}_{i}^{k}$ at stage $k$, $P_{i}$ believes with a probability $b^{k}_{i}(\theta_{j}|{I}_{i}^{k})$ that $P_{j}$ is of type $\theta_{j}\in\Theta_{j}$. The initial belief $b^{0}_{i}:\Theta_{i}\mapsto\bigtriangleup\Theta_{j},\forall i,j\in\{1,2\},j\neq i$, is formed based on an imperfect detection, side-channel information or the statistic estimation resulted from past experiences.

If the system has a perfect recall $\mathcal{I}_{i}^{k}=\mathcal{H}^{k}\times\Theta_{i}$, then players can update their beliefs according to the Bayesian rule:

[TABLE]

Here, $P_{i}$ updates the belief ${b}^{k}_{i}$ based on the observation of the action $a_{i}^{k},a_{j}^{k}$. When the denominator is [math], the history $h^{k+1}$ is not reachable from $h^{k}$, and a Bayesian update does not apply. In this case, we let ${b}^{k+1}_{i}(\theta_{j}|h^{k}\cup\{a_{i}^{k},a_{j}^{k}\},\theta_{i}):=b_{i}^{0}(\theta_{j}|\theta_{i})$.

If the information set is taken to be $\mathcal{I}_{i}^{k}=\mathcal{X}^{k}\times\Theta_{i}$ with the Markov property that $\Pr(x^{k+1}|\theta_{j},x^{k},\cdots,x^{1},x^{0},\theta_{i})=\Pr(x^{k+1}|\theta_{j},x^{k},\theta_{i})$, then the Bayesian update between two consequent states is

[TABLE]

The Markov belief update (2) can be regarded as an approximation of (1) using action aggregations. Unlike the history set $\mathcal{H}^{k}$, the dimension of the state set $|\mathcal{X}^{k}|$ does not grow with the number of stages. Hence, the Markov approximation significantly reduces the memory and computational complexity.

2.3 Utility and PBNE

At each stage $k$, $P_{i}$’s stage utility $\bar{J}_{i}^{k}:\mathcal{X}^{k}\times\mathcal{A}_{1}^{k}\times\mathcal{A}_{2}^{k}\times\theta_{1}\times\theta_{2}\times\mathcal{R}\mapsto\mathcal{R}$ depends on both players’ types and actions, the current state ${x}^{k}\in\mathcal{X}^{k}$, and an external noise $w_{i}^{k}\in\mathcal{R}$ with a known probability density function $\varpi_{i}^{k}$. The noise term models unknown or uncontrolled factors that can affect the value of the stage utility. Denote the expected stage utility as $J_{i}^{k}(x^{k},a^{k}_{1},a^{k}_{2},\theta_{1},\theta_{2}):=E_{w_{i}^{k}\sim\varpi_{i}^{k}}\bar{J}_{i}^{k}(x^{k},a^{k}_{1},a^{k}_{2},\theta_{1},\theta_{2},w_{i}^{k}),\forall x^{k},a^{k}_{1},a^{k}_{2},\theta_{1},\theta_{2}$.

Given the type $\theta_{i}\in\Theta_{i}$, the initial state $x^{k_{0}}\in\mathcal{X}^{k_{0}}$, and both players’ strategies $\sigma_{i}^{k_{0}:K}:=[\sigma^{k}_{i}(a_{i}^{k}|{x}^{k},\theta_{i})]_{k=k_{0},\cdots,K}\in\prod_{k=k_{0}}^{K}\Sigma_{i}^{k}$ from stage $k_{0}$ to $K$, we can determine the expected cumulative utility $U_{i}^{k_{0}:K}$ for $P_{i},i\in\{1,2\}$, by taking expectations over the mixed-strategy distributions and the $P_{i}$’s belief on $P_{j}$’s type, i.e.,

[TABLE]

The attacker and the defender use the Bayesian update to reduce their uncertainties on the other player’s type. Since their actions affect the belief update, both players at each stage should optimize their expected cumulative utilities concerning the updated beliefs, which leads to the solution concept of PBNE in Definition 1.

Definition 1

Consider the two-person $K$-stage game with a double-sided incomplete information, a sequence of beliefs $b_{i}^{k},\forall k\in\{0,\cdots,K\}$, an expected cumulative utility $U^{0:K}_{i}$ in (3), and a given scalar $\varepsilon\geq 0$. A sequence of strategies $\sigma_{i}^{*,0:K}\in\prod_{k=0}^{K}\Sigma_{i}^{k}$ is called $\varepsilon$-perfect Bayesian Nash equilibrium for player $i$ if the following two conditions are satisfied.

• C1

: Belief consistency: under the strategy pair $(\sigma_{1}^{*,0:K},\sigma_{2}^{*,0:K})$, each player’s belief $b_{i}^{k}$ at each stage $k=0,\cdots,K$ satisfies (2).

• C2

: Sequential rationality: for all given initial state $x^{k_{0}}\in\mathcal{X}^{k_{0}}$ at every initial stage $k_{0}\in\{0,\cdots,K\}$, $\forall\sigma_{1}^{k_{0}:K}\in\prod_{k=0}^{K}\Sigma_{1}^{k},\forall\sigma_{2}^{k_{0}:K}\in\prod_{k=0}^{K}\Sigma_{2}^{k}$,

[TABLE]

When $\varepsilon=0$, the equilibrium is called Perfect Bayesian Nash Equilibrium (PBNE). ∎

Solving PBNE is challenging. If the type space is discrete and finite, then given each player’s belief at all stages, we can solve the equilibrium strategy satisfying condition C2 via dynamic programming and a bilinear program. Next, we update the belief at each stage based on the computed equilibrium strategy. We iterate the above update on the equilibrium strategy and belief until they satisfy condition C1 as demonstrated in APTjournal . If the type space is continuous, then the Bayesian update can be simplified into a parametric update under the conjugate prior assumption. Next, the parameter after each belief update can be assimilated into the backward dynamic programming of equilibrium strategy with an expanded state space huang2018analysis . Although no iterations are required, the infinite dimension of continuous type space limits the computation to two by two game matrices.

We apply the above framework and analysis to a case study of Tennessee Eastman (TE) process and investigate both players’ multistage utilities under the adversarial and the defensive deception in Fig. 4. Some insights are listed as follows.

First, the defender’s payoffs under type $\theta_{1}^{H}$ can increase as much as $56\%$ than those under type $\theta_{1}^{L}$. Second, the defender and the attacker receive the highest and the lowest payoff, respectively, under the complete information. When the attacker introduces deceptions over his type, the attacker’s utility increases and the system utility decreases. Third, when the defender adopts defensive deceptions to introduce double-sided incomplete information, we find that the decrease of system utilities is reduced by at most $64\%$, i.e., the decrease of system utilities changes from \55,570$to$$35,570$under the internal state and type$\theta_{1}^{H}$. The double-sided incomplete information also brings lower utilities to the attacker than the one-sided adversarial deception. However, the system utility under the double-sided deception is still less than the complete information case, which concludes that acquiring complete information of the adversarial user is the most effective defense. However, if the complete information cannot be obtained, the defender can mitigate her loss by introducing defensive deceptions.

3 Distributed Learning for Uncertain Payoffs

In the previous section, we study known attacks and systems that adopt cyber deception to conceal their types. We assume common knowledge of the prior probability distribution of the unknown type, and also a common observation of either the action history or the state at each stage. Thus, each player can use Bayesian learning to reduce the other player’s type uncertainty.

In this section, we consider unknown attacks in the MTD game stated in zhu2013game where each player has no information on the past actions of the other player, and the payoff functions are subject to noises and disturbances with unknown statistical characteristics. Without information sharing between players, the learning is distributed.

3.1 Static Game Model of MTD

We consider a system of $N$ layers yet focus on the static game at layer $l\in\{1,2,\cdots,N\}$ because the technique can be employed at each layer of the system independently. At layer $l$, $\mathcal{V}_{l}:=\{v_{l,1},v_{l,2},\cdots,v_{l,n_{l}}\}$ is the set of $n_{l}$ system vulnerabilities that an attacker can exploit to compromise the system. Instead of a static configuration at layer $l$, the defender can choose to change her configuration from a finite set of $m_{l}$ feasible configurations $\mathcal{C}_{l}:=\{c_{l,1},c_{l,2},\cdots,c_{l,m_{l}}\}$. Different configurations result in different subsets of vulnerabilities among $\mathcal{V}_{l}$, which are characterized by the vulnerability map $\pi_{l}:\mathcal{C}_{l}\rightarrow 2^{\mathcal{V}_{l}}$. We call $\pi_{l}(c_{l,j})$ the attack surface at stage $l$ under configuration $c_{l,j}$.

Suppose that for each vulnerability $v_{l,j}$, the attacker can take a corresponding attack $a_{l,j}=\gamma_{l}(v_{l,j})$ from the action set $\mathcal{A}_{l}:=\{a_{l,1},a_{l,2},\cdots,a_{l,n_{l}}\}$. Attack action $a_{l,j}$ is only effective and incurs a bounded cost $D_{ij}\in\mathbb{R}_{+}$ when the vulnerability $v_{l,j}=\gamma_{l}^{-1}(a_{l,j})$ exists in the current attack surface $\pi_{l}(c_{l,k})$. Thus, the damage caused by the attacker at stage $l$ can be represented as

[TABLE]

Since vulnerabilities are inevitable in a modern computing system, we can randomize the configuration and make it difficult for the attacker to learn and locate the system vulnerability, which naturally leads to the mixed strategy equilibrium solution concept of the game. At layer $l$, the defender’s strategy $\mathbf{f}_{l}=\{f_{l,1},f_{l,2},\cdots,f_{l,m_{l}}\}\in\bigtriangleup\mathcal{C}_{l}$ assigns probability $f_{l,j}\in[0,1]$ to configuration $c_{l,j}$ while the attacker’s strategy $\mathbf{g}_{l}:=\{g_{l,1},g_{l,2},\cdots,g_{l,n_{l}}\}\in\bigtriangleup\mathcal{A}_{l}$ assigns probability $g_{l,i}\in[0,1]$ to attack action $a_{l,i}$. The zero-sum game possesses a mixed strategy saddle-point equilibrium (SPE) $(\mathbf{f}_{l}^{*}\in\bigtriangleup\mathcal{C}_{l},\mathbf{g}_{l}^{*}\in\bigtriangleup\mathcal{A}_{l}),$ and a unique game value $\mathbbm{r}(\mathbf{f}_{l}^{*},\mathbf{g}_{l}^{*})$, i.e.,

[TABLE]

where the expected cost $\mathbbm{r}_{l}$ is given by

[TABLE]

We illustrate the multistage MTD game in Fig. 5 and focus on the first layer with two available configurations $\mathcal{C}_{1}:=\{c_{1,1},c_{1,2}\}$ in the blue box. Configuration $c_{1,1}$ in Fig. 5(a) has an attack surface $\pi_{1}(c_{1,1})=\{v_{1,1},v_{1,2}\}$ while configuration $c_{1,2}$ in Fig. 5(b) reveals two vulnerabilities $v_{1,2},v_{1,3}\in\pi_{1}(c_{1,2})$. Then, if the attacker takes action $a_{1,1}$ and the defender changes the configuration from $c_{1,1}$ to $c_{1,2}$, the attack is deterred at the first layer.

3.2 Distributed Learning

In practical cybersecurity domain, the payoff function $r_{l}$ is subjected to noises of unknown distributions. Then, each player reduces the payoff uncertainty by repeatedly observing the payoff realizations during the interaction with the other player. We use subscript $t$ to denote the strategy or cost at time $t$.

There is no communication at any time between two agents due to the non-cooperative environment, and the configuration and attack action are kept private, i.e., each player cannot observe the other player’s action. Thus, each player independently chooses action $\mathbbm{c}_{l,t}\in\mathcal{C}_{l}$ or $\mathbbm{a}_{l,t}\in\mathcal{A}_{l}$ to estimate the average risk of the system $\hat{r}_{l,t}^{S}:\mathcal{C}_{l}\rightarrow\mathbb{R}_{+}$ and $\hat{r}_{l,t}^{A}:\mathcal{A}_{l}\rightarrow\mathbb{R}_{+}$ at layer $l$. Based on the estimated average risk $\hat{r}_{l,t}^{S}$ and the previous policy $\mathbf{f}_{l,t}$, the defender can obtain her updated policy $\mathbf{f}_{l,t+1}$. Likewise, the attacker can also update his policy $\mathbf{g}_{l,t+1}$ based on $\hat{r}_{l,t}^{A}$ and $\mathbf{g}_{l,t}$. The new policy pair $(\mathbf{f}_{l,t+1},\mathbf{g}_{l,t+1})$ determines the next payoff sample. The entire distributed learning feedback loop is illustrated in Fig. 6 where we distinguish the adversarial and defensive learning in red and green, respectively.

In particular, players update their estimated average risks based on the payoff sample $r_{l,t}$ under the chosen action pair $(\mathbbm{c}_{l,t},\mathbbm{a}_{l,t})$ as follows. Let $\mu_{t}^{S}$ and $\mu_{t}^{A}$ be the payoff learning rate for the system and attacker, respectively.

[TABLE]

The indicators in (8) mean that both players only update the estimate average risk of the current action.

Security versus Usability

Frequent configuration changes may achieve the complete security yet also decrease the system usability. To quantify the tradeoff between the security and the usability, we introduce the switching cost of policy from $\mathbf{f}_{l,t}$ to $\mathbf{f}_{l,t+1}$ as their entropy:

[TABLE]

Then, the total cost at time $t$ combines the expected cost with the entropy penalty in a ratio of $\epsilon_{l,t}^{S}$. When $\epsilon_{l,t}^{S}$ is high, the policy changes less and is more usable, yet may cause a large loss and be less rational.

[TABLE]

A similar learning cost is introduced for the attacker:

[TABLE]

At any time $t+1$, we are able to obtain the equilibrium strategy $(f_{l,h,t+1},g_{l,h,t+1})$ and game value $(W_{l,t}^{S},W_{l,t}^{A})$ in closed form of the previous strategy and the estimated average risk at time $t$ as follows.

[TABLE]

Learning Dynamics and ODE Counterparts

The closed form of policy leads to the following learning dynamics with learning rates $\lambda_{l,t}^{S},\lambda_{l,t}^{A}\in[0,1]$.

[TABLE]

If $\lambda_{l,t}^{S}=1,\lambda_{l,t}^{A}=1$, (13) is the same as (12). According to the stochastic approximation theory, the convergence of the policy and the average risk requires the learning rates $\lambda_{l,t}^{A},\lambda_{l,t}^{S},\mu_{l,t}^{A},\mu_{l,t}^{S}$ to satisfy the regular condition of convergency in Definition 2.

Definition 2

A number sequence $\{x_{t}\},t=1,2,\cdots$, is said to satisfy the regular condition of convergency if

[TABLE]

∎

The coupled dynamics of the payoff learning (8) and policy learning(13) converge to their Ordinary Differential Equations (ODEs) counterparts in system dynamics (15) and attacker dynamics (16), respectively. Let $e_{c_{l,h}}\in\bigtriangleup\mathcal{C}_{l},e_{a_{l,h}}\in\bigtriangleup\mathcal{A}_{l}$ be vectors of proper dimensions with the $h$-th entry being $1$ and others being [math].

[TABLE]

[TABLE]

We can show that the SPE of the game is the steady state of the ODE dynamics in (15), (16), and the interior stationary points of the dynamics are the SPE of the game zhu2013game .

Heterogeneous and Hybrid Learning

The entropy regulation terms in (10) and (11) result in a closed form of strategies and learning dynamics in (13). Without the closed form, distributed learners can adopt general learning schemes which combine the payoff and the strategy update as stated in zhu2010heterogeneous . Specifically, algorithm CRL0 mimics the replicator dynamics and updates the strategy according to the current sample value of the utility. On the other hand, algorithm CRL1 updates the strategy according to a soft-max function of the estimated utilities so that the most rewarding policy get reinforced and will be picked with a higher probability. The first algorithm is robust yet inefficient, and the second one is fragile yet efficient. Moreover, players are not obliged to adopt the same learning scheme at different time. The heterogeneous learning focuses on different players adopting different learning schemes zhu2010heterogeneous , while hybrid learning means that players can choose different learning schemes at different times based on their rationalities and preferences zhu2011distributed . According to stochastic approximation techniques, these learning schemes with random updates can be studied using their deterministic ODE counterparts.

4 Reinforcement Learning for Uncertain Environments

This section considers uncertainties on the entire environment, i.e., the state transition, the sojourn time, and the investigation payoff, in the active defense scenario of the honeypot engagement huangHoneypot . We use the Semi-Markov Decision Process (SMDP) to capture these environmental uncertainties in the continuous time system. Although the attacker’s duration time is continuous at each honeypot, the defender’s engagement action is applied at a discrete time epoch. Based on the observed samples at each decision epoch, the defender can estimate the environment elements determined by attackers’ characteristics, and use reinforcement learning methods to obtain the optimal policy. We plot the entire feedback learning structure in Fig. 7. Since the attacker should not identify the existence of the honeypot and the defender’s engagement actions, he will not take actions to jeopardize the learning.

4.1 Honeypot Network and SMDP Model

The honeypots form a network to emulate a production system. From an attacker’s viewpoint, two network structures are the same as shown in Fig. 8.

Based on the network topology, we introduce the continuous-time infinite-horizon discounted SMDPs, which can be summarized by the tuple $\{t\in[0,\infty),\mathcal{S},\mathcal{A}({s_{j}}),tr(s_{l}|s_{j},a_{j}),\allowbreak z(\cdot|s_{j},a_{j},s_{l}),r^{\gamma}(s_{j},a_{j},s_{l}),\gamma\in[0,\infty)\}$. We illustrate each element of the tuple through a $13$-state example in Fig. 9.

Each node in Fig. 9 represents a state $s_{i}\in\mathcal{S},i\in\{1,2,\cdots,13\}$. At time $t\in[0,\infty)$, the attacker is either at one of the honeypot node denoted by state $s_{i}\in\mathcal{S},i\in\{1,2,\cdots,11\}$, at the normal zone $s_{12}$, or at a virtual absorbing state $s_{13}$ once attackers are ejected or terminate on their own. At each state $s_{i}\in\mathcal{S}$, the defender can choose an action $a_{i}\in\mathcal{A}(s_{i})$. For example, at honeypot nodes, the defender can conduct action $a_{E}$ to eject the attacker, action $a_{P}$ to purely record the attacker’s activities, low-interactive action $a_{L}$, or high-interactive action $a_{H}$, i.e., $\mathcal{A}(s_{i}):=\{a_{E},a_{P},a_{L},a_{H}\},i\in\{1,\cdots,\allowbreak N\}$. The high-interactive action is costly to implement yet can both increases the probability of a longer sojourn time at honeypot $n_{i}$, and reduces the probability of attackers penetrating the normal system from $n_{i}$ if connected. If the attacker resides in the normal zone either from the beginning or later through the pivot honeypots, the defender can choose either action $a_{E}$ to eject the attacker immediately, or action $a_{A}$ to attract the attacker to the honeynet by generating more deceptive inbound and outbound traffics in the honeynet, i.e., $\mathcal{A}(s_{12}):=\{a_{E},a_{A}\}$.

Based on the current state $s_{j}\in\mathcal{S}$ and the defender’s action $a_{j}\in\mathcal{A}(s_{j})$, the attacker transits to state $s_{l}\in\mathcal{S}$ with probability $tr(s_{l}|s_{j},a_{j})$ and the sojourn time at state $s_{j}$ is a continuous random variable with probability density $z(\cdot|s_{j},a_{j},s_{l})$. Once the attacker arrives at a new honeypot $n_{i}$, the defender dynamically applies an interaction action at honeypot $n_{i}$ from $\mathcal{A}(s_{i})$ and keeps interacting with the attacker until she transits to the next honeypot. If the defender changes the action before the transition, the attacker may be able to detect the change and become aware of the honeypot. Since the decision is made at the time of transition, we can transform the above continuous time model on horizon $t\in[0,\infty)$ into a discrete decision model at decision epoch $k\in\{0,1,\cdots,\infty\}$. The time of the attacker’s $k^{th}$ transition is denoted by a random variable $T^{k}$, the landing state is denoted as $s^{k}\in\mathcal{S}$, and the adopted action after arriving at $s^{k}$ is denoted as $a^{k}\in\mathcal{A}(s^{k})$.

The defender gains an investigation reward by engaging and analyzing the attacker in the honeypot. To simplify the notation, we segment the investigation reward during time $t\in[0,\infty)$ into ones at discrete decision epochs $T^{k},k\in\{0,1,\cdots,\infty\}$. When $\tau\in[T^{k},T^{k+1}]$ amount of time elapses at stage $k$, the defender’s investigation reward $r(s^{k},a^{k},s^{k+1},T^{k},T^{k+1},\tau)=r_{1}(s^{k},a^{k},s^{k+1})\mathbf{1}_{\{\tau=0\}}+r_{2}(s^{k},a^{k},T^{k},T^{k+1},\tau)$, at time $\tau$ of stage $k$, is the sum of two parts. The first part is the immediate cost of applying engagement action $a^{k}\in\mathcal{A}(s^{k})$ at state $s^{k}\in\mathcal{S}$ and the second part is the reward rate of threat information acquisition minus the cost rate of persistently generating deceptive traffics. Due to the randomness of the attacker’s behavior, the information acquisition can also be random, thus the actual reward rate $r_{2}$ is perturbed by an additive zero-mean noise $w_{r}$. As the defender spends longer time interacting with attackers, investigating their behaviors and acquires better understandings of their targets and TTPs, less new information can be extracted. In addition, the same intelligence becomes less valuable as time elapses due to the timeliness. Thus, we use a discounted factor of $\gamma\in[0,\infty)$ to penalize the decreasing value of the investigation reward as time elapses.

The defender aims at a policy $\pi\in\Pi$ which maps state $s^{k}\in\mathcal{S}$ to action $a^{k}\in\mathcal{A}(s^{k})$ to maximize the long-term expected utility starting from state $s^{0}$, i.e.,

[TABLE]

At each decision epoch, the value function $v(s^{0})=\sup_{\pi\in\Pi}u(s^{0},\pi)$ can be represented by dynamic programming, i.e.,

[TABLE]

We assume a constant reward rate $r_{2}(s^{k},a^{k},T^{k},T^{k+1},\tau)=\bar{r}_{2}(s^{k},a^{k})$ for simplicity. Then, (18) can be transformed into an equivalent MDP form, i.e., $\forall s^{0}\in\mathcal{S}$,

[TABLE]

where ${z^{\gamma}}(s^{0},a^{0},s^{1}):=\int_{0}^{\infty}e^{-\gamma\tau}z(\tau|s^{0},a^{0},s^{1})d\tau\in[0,1]$ is the Laplace transform of the sojourn probability density $z(\tau|s^{0},a^{0},s^{1})$ and the equivalent reward $r^{\gamma}(s^{0},a^{0},s^{1})\allowbreak:=r_{1}(s^{0},a^{0},s^{1})+\frac{\bar{r}_{2}(s^{0},a^{0})}{\gamma}(1-z^{\gamma}(s^{0},a^{0},s^{1}))\in[-m_{c},m_{c}]$ is assumed to be bounded by a constant $m_{c}$.

Definition 3

There exists constants $\theta\in(0,1)$ and $\delta>0$ such that

[TABLE]

∎

The right-hand side of (18) is a contraction mapping under the regulation condition in Definition 3. Then, we can find the unique optimal policy $\pi^{*}=arg\max_{\pi\in\Pi}u(s^{0},\pi)$ by value iteration, policy iteration or linear programming. Fig. 9 illustrates the optimal policy and the state value by the color and the size of the node, respectively. In the example scenario, the honeypot of database $n_{10}$ and sensors $n_{11}$ are the main and secondary targets of the attacker, respectively. Thus, defenders can obtain a higher investigation reward when they manage to engage the attacker in these two honeypot nodes with a larger probability and for a longer time. However, instead of naively adopting high interactive actions, a savvy defender also balances the high implantation cost of $a_{H}$. Our quantitative results indicate that the high interactive action should only be applied at $n_{10}$ to be cost-effective. On the other hand, although the bridge nodes $n_{1},n_{2},n_{8}$ which connect to the normal zone $n_{12}$ do not contain higher investigation rewards than other nodes, the defender still takes action $a_{L}$ at these nodes. The goal is to either increase the probability of attracting attackers away from the normal zone or reduce the probability of attackers penetrating the normal zone from these bridge nodes.

4.2 Reinforcement Learning of SMDP

The absent knowledge of the attacker’s characteristics results in environmental uncertainty of the investigation reward, the attacker’s transition probability, and the sojourn distribution. We use $Q$-learning algorithm to obtain the optimal engagement policy based on the actual experience of the honeynet interactions, i.e., $\forall\bar{s}^{k}\in\mathcal{S},\forall a^{k}\in\mathcal{A}(\bar{s}^{k})$,

[TABLE]

where $\alpha^{k}(\bar{s}^{k},a^{k})\in(0,1)$ is the learning rate, $\bar{s}^{k},\bar{s}^{k+1}$ are the observed states at stage $k$ and $k+1$, $\bar{r}_{1},\bar{r}_{2}$ is the observed investigation rewards, and $\bar{\tau}^{k}$ is the observed sojourn time at state $s^{k}$. When the learning rate satisfies the condition of convergency in Definition 2, i.e., $\sum_{k=0}^{\infty}\alpha^{k}(s^{k},a^{k})=\infty,\sum_{k=0}^{\infty}(\alpha^{k}(s^{k},a^{k}))^{2}<\infty,\forall s^{k}\in\mathcal{S},\forall a^{k}\in\mathcal{A}(s^{k})$, and all state-action pairs are explored infinitely, $\max_{a^{\prime}\in\mathcal{A}(s^{k})}\allowbreak Q^{\infty}(s^{\infty},a^{\prime})$, in (21) converges to value $v(s^{k})$ with probability $1$.

At each decision epoch $k\in\{0,1,\cdots\}$, the action $a^{k}$ is chosen according to the $\epsilon$-greedy policy, i.e., the defender chooses the optimal action $arg\max_{a^{\prime}\in\mathcal{A}(s^{k})}Q^{k}(s^{k},a^{\prime})$ with a probability $1-\epsilon$, and a random action with a probability $\epsilon$. Note that the exploration rate $\epsilon\in(0,1]$ should not be too small to guarantee sufficient samples of all state-action pairs. The $Q$-learning algorithm under a pure exploration policy $\epsilon=1$ still converges yet at a slower rate.

In our scenario, the defender knows the reward of ejection action $a_{A}$ and $v(s_{13})=0$, thus does not need to explore action $a_{A}$ to learn it. We plot one learning trajectory of the state transition and sojourn time under the $\epsilon$-greedy exploration policy in Fig. 10, where the chosen actions $a_{E},a_{P},a_{L},a_{H}$ are denoted in red, blue, purple, and green, respectively. If the ejection reward is unknown, the defender should be restrictive in exploring $a_{A}$ which terminates the learning process. Otherwise, the defender may need to engage with a group of attackers who share similar behaviors to obtain sufficient samples to learn the optimal engagement policy.

In particular, we choose $\alpha^{k}(s^{k},a^{k})=\frac{k_{c}}{k_{\{s^{k},a^{k}\}}-1+k_{c}},\forall s^{k}\in\mathcal{S},\forall a^{k}\in\mathcal{A}(s^{k})$, to guarantee the asymptotic convergence, where $k_{c}\in(0,\infty)$ is a constant parameter and $k_{\{s^{k},a^{k}\}}\in\{0,1,\cdots\}$ is the number of visits to state-action pair $\{s^{k},a^{k}\}$ up to stage $k$. We need to choose a proper value of $k_{c}$ to guarantee a good numerical performance of convergence in finite steps as shown in Fig. 11(a). We shift the green and blue lines vertically to avoid the overlap with the red line and represent the corresponding theoretical values in dotted black lines. If $k_{c}$ is too small as shown in the red line, the learning rate decreases so fast that new observed samples hardly update the $Q$-value and the defender may need a long time to learn the right value. However, if $k_{c}$ is too large as shown in the green line, the learning rate decreases so slow that new samples contribute significantly to the current $Q$-value. It causes a large variation and a slower convergence rate of $\max_{a^{\prime}\in\mathcal{A}(s_{12})}Q^{k}(s_{12},a^{\prime})$.

We show the convergence of the policy and value under $k_{c}=1,\epsilon=0.2$, in the video demo (See URL: https://bit.ly/2QUz3Ok). In the video, the color of each node $n^{k}$ distinguishes the defender’s action $a^{k}$ at state $s^{k}$ and the size of the node is proportional to $\max_{a^{\prime}\in\mathcal{A}(s^{k})}Q^{k}(s^{k},a^{\prime})$ at stage $k$. To show the convergence, we decrease the value of $\epsilon$ gradually to [math] after $5000$ steps. Since the convergence trajectory is stochastic, we run the simulation for $100$ times and plot the mean and the variance of $Q^{k}(s_{12},a_{P})$ of state $s_{12}$ under the optimal policy $\pi(s_{12})=a_{P}$ in Fig. 11. The mean in red converges to the theoretical value in about $400$ steps and the variance in blue reduces dramatically as step $k$ increases.

5 Conclusion and Discussion

This chapter has introduced three defense schemes, i.e., defensive deception to detect and counter adversarial deception, feedback-driven Moving Target Defense (MTD) to increase the attacker’s probing and reconnaissance costs, and adaptive honeypot engagement to gather fundamental threat information. These schemes satisfy the Principle of 3A Defense as they actively protect the system prior to the attack damages, provide strategic defenses autonomously, and apply learning to adapt to uncertainty and changes. These schemes possess three progressive levels of information restrictions, which lead to different strategic learning schemes to estimate the parameter, the payoff, and the environment. All these learning schemes, however, have a feedback loop to sense samples, estimate the unknowns, and take actions according to the estimate. Our work lays a solid foundation for strategic learning in active, adaptive, autonomous defenses under incomplete information and leads to the following challenges and future directions.

First, multi-agent learning in non-cooperative environments is challenging due to the coupling and interaction between these heterogeneous agents. The learning results depend on all involving agents yet other players’ behaviors, levels of rationality, and learning schemes are not controllable and may change abruptly. Moreover, as attackers become aware of the active defense techniques and the learning scheme under incomplete information, the savvy attacker can attempt to interrupt the learning process. For example, attackers may sacrifice their immediate rewards and take incomprehensible actions instead so that the defender learns incorrect attack characteristics. The above challenges motivate robust learning methods under non-cooperative and even adversarial environments.

Second, since the learning process is based on samples from real interactions, the defender needs to concern the system safety and security during the learning period, while in the same time, attempts to achieve more accurate learning results of the attack’s characteristics. Moreover, since the learning under non-cooperative and adversarial environments may terminate unpredictably at any time, the asymptotic convergence would not be critical for security. The defender needs to care more about the time efficiency of the learning, i.e., how to achieve a sufficiently good estimate in a finite number of steps.

Third, instead of learning from scratch, the defender can attempt to reuse the past experience with attackers of similar behaviors to expedite the learning process, which motivates the investigation of transfer learning in reinforcement learning taylor2009transfer . Some side-channel information may also contribute to the learning to allow agents to learn faster.