Can Developers rely on LLMs for Secure IaC Development?

TL;DR

This paper evaluates GPT-4o and Gemini 2.0 Flash's effectiveness in detecting security smells and generating secure code in Infrastructure as Code development, revealing moderate success and highlighting areas for improvement.

Contribution

It provides an empirical assessment of LLMs' capabilities in secure IaC development, introducing the impact of prompt guidance on security detection and code generation.

Findings

Prompt guidance improves security smell detection from 71% to 78% on Stack Overflow data.

LLMs generate only 7-17% secure scripts in synthetic vulnerability scenarios.

Models perform less effectively on real-world GitHub repositories compared to simplified datasets.

Abstract

We investigated the capabilities of GPT-4o and Gemini 2.0 Flash for secure Infrastructure as Code (IaC) development. For security smell detection, on the Stack Overflow dataset, which primarily contains small, simplified code snippets, the models detected at least 71% of security smells when prompted to analyze code from a security perspective (general prompt). With a guided prompt (adding clear, step-by-step instructions), this increased to 78%.In GitHub repositories, which contain complete, real-world project scripts, a general prompt was less effective, leaving more than half of the smells undetected. However, with the guided prompt, the models uncovered at least 67% of the smells. For secure code generation, we prompted LLMs with 89 vulnerable synthetic scenarios and observed that only 7% of the generated scripts were secure. Adding an explicit instruction to generate secure code increased GPT secure output rate to 17%, while Gemini changed little (8%). These results highlight the need for further research to improve LLMs' capabilities in assisting developers with secure IaC development.