Evaluating the Reliability of Multiple Large Language Models in Risk Assessment: A CIS Controls Based Approach

TL;DR

This study evaluates the performance of large language models in cybersecurity risk assessment, emphasizing the importance of human oversight due to their tendency to underestimate risks.

Contribution

It compares LLMs with human experts in cybersecurity risk assessment, highlighting the necessity of integrating human validation for reliable results.

Findings

LLMs consistently underestimate cybersecurity risks compared to humans.

Human oversight is crucial to ensure accurate risk assessments.

LLMs should be used as complementary tools, not standalone assessors.

Abstract

Proper implementation of technical and administrative controls reinforces an organization's cybersecurity posture and business resilience, reduces risks, and enhances governance, ultimately elevating business maturity. The dynamics of the technological landscape and emerging threats negatively affect the most diverse companies, regardless of their size. This, associated with a global gap in the cybersecurity workforce, imposes enormous challenges and the need for a profound change in how companies respond to threats. Generative Artificial Intelligence from large language models has become an influential tool across various companies, emerging as a viable option to help address those challenges while partially addressing the shortage of skilled labor. Although large language models can help in this scenario, there may be risks, such as generating unreliable or 'hallucinated' content, which could lead people and companies to make bad decisions. Our study proposes integrating human experts into the validation process as a crucial step toward ensuring the proper implementation of technical and administrative controls. Furthermore, we sought to identify how large language models perform in assessing cybersecurity risk scenarios compared to human experts, highlighting the importance of integrating humans and machines in the cybersecurity risk assessment process. Using a questionnaire with risk scenarios, we analyzed responses from 50 human experts. We compared their responses with those of five popular large language models to determine whether it is possible to use only large language models for cybersecurity risk assessment. The results reveal that the large language models consistently underestimated cybersecurity risks compared to human experts, reinforcing the need for human oversight and suggesting that LLMs should be used as complementary tools rather than standalone assessors.