Evaluating the Usability of LLMs in Threat Intelligence Enrichment

TL;DR

This paper evaluates the usability of five large language models for threat intelligence enrichment, identifying key usability issues and providing recommendations to improve their integration and effectiveness for security professionals.

Contribution

It offers a comprehensive usability assessment of LLMs in threat intelligence, highlighting design and reliability challenges and proposing actionable improvements.

Findings

Identified key usability issues in LLM interfaces and error handling.

Provided actionable recommendations for enhancing LLM integration.

Demonstrated potential for improved threat intelligence workflows.

Abstract

Large Language Models (LLMs) have the potential to significantly enhance threat intelligence by automating the collection, preprocessing, and analysis of threat data. However, the usability of these tools is critical to ensure their effective adoption by security professionals. Despite the advanced capabilities of LLMs, concerns about their reliability, accuracy, and potential for generating inaccurate information persist. This study conducts a comprehensive usability evaluation of five LLMs ChatGPT, Gemini, Cohere, Copilot, and Meta AI focusing on their user interface design, error handling, learning curve, performance, and integration with existing tools in threat intelligence enrichment. Utilizing a heuristic walkthrough and a user study methodology, we identify key usability issues and offer actionable recommendations for improvement. Our findings aim to bridge the gap between LLM functionality and user experience, thereby promoting more efficient and accurate threat intelligence practices by ensuring these tools are user-friendly and reliable.