Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis

TL;DR

This paper compares open-source vulnerability scanners for conversational LLMs, highlighting their features, limitations, and reliability issues, and introduces a labelled dataset to improve future vulnerability detection.

Contribution

It provides a comprehensive comparison of prominent LLM vulnerability scanners, identifies reliability gaps, and offers a labelled dataset to aid future research and development.

Findings

Significant reliability issues in current scanners

Unifying principles of scanner design identified

A preliminary labelled dataset introduced

Abstract

This report presents a comparative analysis of open-source vulnerability scanners for conversational large language models (LLMs). As LLMs become integral to various applications, they also present potential attack surfaces, exposed to security risks such as information leakage and jailbreak attacks. Our study evaluates prominent scanners - Garak, Giskard, PyRIT, and CyberSecEval - that adapt red-teaming practices to expose these vulnerabilities. We detail the distinctive features and practical use of these scanners, outline unifying principles of their design and perform quantitative evaluations to compare them. These evaluations uncover significant reliability issues in detecting successful attacks, highlighting a fundamental gap for future development. Additionally, we contribute a preliminary labelled dataset, which serves as an initial step to bridge this gap. Based on the above, we provide strategic recommendations to assist organizations choose the most suitable scanner for their red-teaming needs, accounting for customizability, test suite comprehensiveness, and industry-specific use cases.