CYGENT: A cybersecurity conversational agent with log summarization powered by GPT-3

TL;DR

CYGENT is a GPT-3.5 turbo-powered conversational agent designed for cybersecurity, capable of log summarization, event detection, and providing security insights, thereby aiding system administrators in managing complex cyber environments.

Contribution

This work fine-tunes GPT-3.5 turbo for cybersecurity tasks and demonstrates its superior performance in log summarization and analysis compared to other LLMs.

Findings

GPT-3.5 turbo achieved over 97% BERTscore in log summarization.

Davinci GPT-3 outperformed other tested LLMs in log analysis.

CodeT5-base-multi-sum shows potential as an offline log summarization model.

Abstract

In response to the escalating cyber-attacks in the modern IT and IoT landscape, we developed CYGENT, a conversational agent framework powered by GPT-3.5 turbo model, designed to aid system administrators in ensuring optimal performance and uninterrupted resource availability. This study focuses on fine-tuning GPT-3 models for cybersecurity tasks, including conversational AI and generative AI tailored specifically for cybersecurity operations. CYGENT assists users by providing cybersecurity information, analyzing and summarizing uploaded log files, detecting specific events, and delivering essential instructions. The conversational agent was developed based on the GPT-3.5 turbo model. We fine-tuned and validated summarizer models (GPT3) using manually generated data points. Using this approach, we achieved a BERTscore of over 97%, indicating GPT-3's enhanced capability in summarizing log files into human-readable formats and providing necessary information to users. Furthermore, we conducted a comparative analysis of GPT-3 models with other Large Language Models (LLMs), including CodeT5-small, CodeT5-base, and CodeT5-base-multi-sum, with the objective of analyzing log analysis techniques. Our analysis consistently demonstrated that Davinci (GPT-3) model outperformed all other LLMs, showcasing higher performance. These findings are crucial for improving human comprehension of logs, particularly in light of the increasing numbers of IoT devices. Additionally, our research suggests that the CodeT5-base-multi-sum model exhibits comparable performance to Davinci to some extent in summarizing logs, indicating its potential as an offline model for this task.