How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models

TL;DR

This large-scale study evaluates the security of AI-generated C code from various LLMs, revealing that over 62% of generated programs contain vulnerabilities, with minor differences across models, emphasizing the need for validation.

Contribution

The paper introduces the FormAI-v2 dataset and provides a comprehensive comparison of multiple state-of-the-art LLMs regarding their tendency to produce vulnerable code.

Findings

Over 62% of generated programs are vulnerable.

Models show similar vulnerability patterns with minor differences.

Formal verification reduces false positives and negatives in vulnerability detection.

Abstract

This study compares state-of-the-art Large Language Models (LLMs) on their tendency to generate vulnerabilities when writing C programs using a neutral zero-shot prompt. Tihanyi et al. introduced the FormAI dataset at PROMISE'23, featuring 112,000 C programs generated by GPT-3.5-turbo, with over 51.24% identified as vulnerable. We extended that research with a large-scale study involving 9 state-of-the-art models such as OpenAI's GPT-4o-mini, Google's Gemini Pro 1.0, TII's 180 billion-parameter Falcon, Meta's 13 billion-parameter Code Llama, and several other compact models. Additionally, we introduce the FormAI-v2 dataset, which comprises 331 000 compilable C programs generated by these LLMs. Each program in the dataset is labeled based on the vulnerabilities detected in its source code through formal verification, using the Efficient SMT-based Context-Bounded Model Checker (ESBMC). This technique minimizes false positives by providing a counterexample for the specific vulnerability and reduces false negatives by thoroughly completing the verification process. Our study reveals that at least 62.07% of the generated programs are vulnerable. The differences between the models are minor, as they all show similar coding errors with slight variations. Our research highlights that while LLMs offer promising capabilities for code generation, deploying their output in a production environment requires proper risk assessment and validation.