AnomalyExplainer Explainable AI for LLM-based anomaly detection using BERTViz and Captum

TL;DR

This paper introduces an explainable AI framework for LLM-based anomaly detection that combines visualization tools and natural language reports to improve transparency, speed, and user trust in cybersecurity applications.

Contribution

It presents a novel framework integrating BERTViz and Captum for explainability in LLM-based anomaly detection, with comparative analysis of model performance and user feedback validation.

Findings

RoBERTa achieves 99.6% accuracy in anomaly detection

The framework enhances user understanding and trust in AI decisions

RoBERTa outperforms Falcon-7B and DeBERTa on LogHub dataset

Abstract

Conversational AI and Large Language Models (LLMs) have become powerful tools across domains, including cybersecurity, where they help detect threats early and improve response times. However, challenges such as false positives and complex model management still limit trust. Although Explainable AI (XAI) aims to make AI decisions more transparent, many security analysts remain uncertain about its usefulness. This study presents a framework that detects anomalies and provides high-quality explanations through visual tools BERTViz and Captum, combined with natural language reports based on attention outputs. This reduces manual effort and speeds up remediation. Our comparative analysis showed that RoBERTa offers high accuracy (99.6 %) and strong anomaly detection, outperforming Falcon-7B and DeBERTa, as well as exhibiting better flexibility than large-scale Mistral-7B on the HDFS dataset from LogHub. User feedback confirms the chatbot's ease of use and improved understanding of anomalies, demonstrating the ability of the developed framework to strengthen cybersecurity workflows.