Automated Security Response through Online Learning with Adaptive Conjectures

TL;DR

This paper introduces Conjectural Online Learning (COL), a novel adaptive method for automated security response that accounts for model uncertainty and improves strategy convergence in dynamic, adversarial IT environments.

Contribution

The paper develops COL, a new online learning approach that incorporates Bayesian conjectures and rollout strategies, addressing model misspecification in security game scenarios.

Findings

COL converges to best-fit conjectures effectively.

COL outperforms existing reinforcement learning in convergence speed.

Testbed results demonstrate adaptive and effective security strategies.

Abstract

We study automated security response for an IT infrastructure and formulate the interaction between an attacker and a defender as a partially observed, non-stationary game. We relax the standard assumption that the game model is correctly specified and consider that each player has a probabilistic conjecture about the model, which may be misspecified in the sense that the true model has probability 0. This formulation allows us to capture uncertainty and misconception about the infrastructure and the intents of the players. To learn effective game strategies online, we design Conjectural Online Learning (COL), a novel method where a player iteratively adapts its conjecture using Bayesian learning and updates its strategy through rollout. We prove that the conjectures converge to best fits, and we provide a bound on the performance improvement that rollout enables with a conjectured model. To characterize the steady state of the game, we propose a variant of the Berk-Nash equilibrium. We present COL through an advanced persistent threat use case. Testbed evaluations show that COL produces effective security strategies that adapt to a changing environment. We also find that COL enables faster convergence than current reinforcement learning techniques.