More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool

TL;DR

Implementing explainable AI in cybersecurity workflows reveals significant challenges, including interpretability issues for non-experts and limitations of current techniques in real-time contexts, highlighting the need for higher-level explanations and potential solutions like LLMs.

Contribution

This paper provides empirical insights into the practical challenges of integrating XAI into cybersecurity workflows and discusses potential technological solutions.

Findings

Saliency explanations are hard to interpret for non-experts

Current XAI techniques are less effective in real-time workflows

Emerging LLMs may help address existing XAI challenges

Abstract

We share observations and challenges from an ongoing effort to implement Explainable AI (XAI) in a domain-specific workflow for cybersecurity analysts. Specifically, we briefly describe a preliminary case study on the use of XAI for source code classification, where accurate assessment and timeliness are paramount. We find that the outputs of state-of-the-art saliency explanation techniques (e.g., SHAP or LIME) are lost in translation when interpreted by people with little AI expertise, despite these techniques being marketed for non-technical users. Moreover, we find that popular XAI techniques offer fewer insights for real-time human-AI workflows when they are post hoc and too localized in their explanations. Instead, we observe that cyber analysts need higher-level, easy-to-digest explanations that can offer as little disruption as possible to their workflows. We outline unaddressed gaps in practical and effective XAI, then touch on how emerging technologies like Large Language Models (LLMs) could mitigate these existing obstacles.