Out-of-Distribution Detection for Neurosymbolic Autonomous Cyber Agents

TL;DR

This paper introduces an out-of-distribution detection method using probabilistic neural networks to improve the trustworthiness of reinforcement learning-based cyber agents in detecting unfamiliar or anomalous situations.

Contribution

It presents a novel OOD monitoring algorithm integrated with a neurosymbolic cyber agent, enhancing reliability in cyber defense scenarios.

Findings

Effective detection of OOD situations demonstrated in simulations

Improved trustworthiness of autonomous cyber agents shown

Robustness against adversarial strategies confirmed

Abstract

Autonomous agents for cyber applications take advantage of modern defense techniques by adopting intelligent agents with conventional and learning-enabled components. These intelligent agents are trained via reinforcement learning (RL) algorithms, and can learn, adapt to, reason about and deploy security rules to defend networked computer systems while maintaining critical operational workflows. However, the knowledge available during training about the state of the operational network and its environment may be limited. The agents should be trustworthy so that they can reliably detect situations they cannot handle, and hand them over to cyber experts. In this work, we develop an out-of-distribution (OOD) Monitoring algorithm that uses a Probabilistic Neural Network (PNN) to detect anomalous or OOD situations of RL-based agents with discrete states and discrete actions. To demonstrate the effectiveness of the proposed approach, we integrate the OOD monitoring algorithm with a neurosymbolic autonomous cyber agent that uses behavior trees with learning-enabled components. We evaluate the proposed approach in a simulated cyber environment under different adversarial strategies. Experimental results over a large number of episodes illustrate the overall efficiency of our proposed approach.