Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination

TL;DR

This paper introduces a lightweight incident response planning method using a fine-tuned large language model that reduces hallucinations, improves response times, and is practical for deployment on standard hardware.

Contribution

It presents a novel incident response planning approach combining fine-tuning, retrieval, and lookahead, with provable hallucination bounds and practical efficiency.

Findings

Achieves up to 22% shorter recovery times than frontier LLMs.

Generalizes across various incident types and response actions.

Reduces hallucination probability with increased planning time.

Abstract

Timely and effective incident response is key to managing the growing frequency of cyberattacks. However, identifying the right response actions for complex systems is a major technical challenge. A promising approach to mitigate this challenge is to use the security knowledge embedded in large language models (LLMs) to assist security operators during incident handling. Recent research has demonstrated the potential of this approach, but current methods are mainly based on prompt engineering of frontier LLMs, which is costly and prone to hallucinations. We address these limitations by presenting a novel way to use an LLM for incident response planning with reduced hallucination. Our method includes three steps: fine-tuning, information retrieval, and lookahead planning. We prove that our method generates response plans with a bounded probability of hallucination and that this probability can be made arbitrarily small at the expense of increased planning time under certain assumptions. Moreover, we show that our method is lightweight and can run on commodity hardware. We evaluate our method on logs from incidents reported in the literature. The experimental results show that our method a) achieves up to 22% shorter recovery times than frontier LLMs and b) generalizes to a broad range of incident types and response actions.