Autonomous Cyber Defense Introduces Risk: Can We Manage the Risk?

TL;DR

This paper discusses the potential benefits and risks of autonomous cyber defenses, emphasizing the need for trust and careful management of unintended consequences in deploying machine learning-based security agents.

Contribution

It highlights the dual nature of autonomous cyber defenses, exploring how ML can improve security while also introducing new risks that require careful oversight.

Findings

Autonomous defenses can adapt to new threats using machine learning.

Unintended harm from autonomous actions poses significant risks.

Trust in autonomous cyber defenses is crucial for their effective deployment.

Abstract

From denial-of-service attacks to spreading of ransomware or other malware across an organization's network, it is possible that manually operated defenses are not able to respond in real time at the scale required, and when a breach is detected and remediated the damage is already made. Autonomous cyber defenses therefore become essential to mitigate the risk of successful attacks and their damage, especially when the response time, effort and accuracy required in those defenses is impractical or impossible through defenses operated exclusively by humans. Autonomous agents have the potential to use ML with large amounts of data about known cyberattacks as input, in order to learn patterns and predict characteristics of future attacks. Moreover, learning from past and present attacks enable defenses to adapt to new threats that share characteristics with previous attacks. On the other hand, autonomous cyber defenses introduce risks of unintended harm. Actions arising from autonomous defense agents may have harmful consequences of functional, safety, security, ethical, or moral nature. Here we focus on machine learning training, algorithmic feedback, and algorithmic constraints, with the aim of motivating a discussion on achieving trust in autonomous cyber defenses.