Does Johnny Get the Message? Evaluating Cybersecurity Notifications for Everyday Users

TL;DR

This paper introduces a framework for evaluating cybersecurity alerts generated by large language models, focusing on their clarity, accuracy, and usefulness for everyday users, and demonstrates its application through various use cases.

Contribution

The paper presents the Human-Centered Security Alert Evaluation Framework (HCSAEF), a novel method for assessing LLM-generated cybersecurity notifications for user comprehension and reliability.

Findings

HCSAEF effectively differentiates alerts based on intuitiveness, urgency, and correctness.

Prompt design and model choice significantly impact notification quality.

HCSAEF supports comparison and improvement of cybersecurity alerts for end users.

Abstract

Due to the increasing presence of networked devices in everyday life, not only cybersecurity specialists but also end users benefit from security applications such as firewalls, vulnerability scanners, and intrusion detection systems. Recent approaches use large language models (LLMs) to rewrite brief, technical security alerts into intuitive language and suggest actionable measures, helping everyday users understand and respond appropriately to security risks. However, it remains an open question how well such alerts are explained to users. LLM outputs can also be hallucinated, inconsistent, or misleading. In this work, we introduce the Human-Centered Security Alert Evaluation Framework (HCSAEF). HCSAEF assesses LLM-generated cybersecurity notifications to support researchers who want to compare notifications generated for everyday users, improve them, or analyze the capabilities of different LLMs in explaining cybersecurity issues. We demonstrate HCSAEF through three use cases, which allow us to quantify the impact of prompt design, model selection, and output consistency. Our findings indicate that HCSAEF effectively differentiates generated notifications along dimensions such as intuitiveness, urgency, and correctness.