LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models

TL;DR

This paper evaluates the security of C/C++ code generated by large language models, revealing prevalent vulnerabilities and emphasizing the need for caution and further research in automated code generation.

Contribution

It provides an empirical analysis of vulnerabilities in LLM-generated C/C++ code, mapping CWEs to CVEs and highlighting security concerns.

Findings

High prevalence of CWEs in AI-generated code

Static analysis reveals significant security vulnerabilities

Calls for improved safety measures in code generation

Abstract

The security of code generated by large language models (LLMs) is a significant concern, as studies indicate that such code often contains vulnerabilities and lacks essential defensive programming constructs. This work focuses on examining and evaluating the security of LLM-generated code, particularly in the context of C/C++. We categorized known vulnerabilities using the Common Weakness Enumeration (CWE) and, to study their criticality, mapped them to CVEs. We used ten different LLMs for code generation and analyzed the outputs through static analysis. The amount of CWEs present in AI-generated code is concerning. Our findings highlight the need for developers to be cautious when using LLM-generated code. This study provides valuable insights to advance automated code generation and encourage further research in this domain.