Can LLMs Patch Security Issues?

TL;DR

This paper introduces Feedback-Driven Security Patching (FDSP), a method that uses static code analysis and LLMs to automatically identify and fix security vulnerabilities in generated code, improving safety in critical applications.

Contribution

The paper presents FDSP, a novel approach combining static analysis and LLMs for automatic security patching, along with a large dataset for real-world application testing.

Findings

FDSP outperforms prior self-feedback methods by up to 17.6%

Introduces PythonSecurityEval dataset covering diverse real-world applications

Demonstrates effective automatic vulnerability detection and fixing

Abstract

Large Language Models (LLMs) have shown impressive proficiency in code generation. Unfortunately, these models share a weakness with their human counterparts: producing code that inadvertently has security vulnerabilities. These vulnerabilities could allow unauthorized attackers to access sensitive data or systems, which is unacceptable for safety-critical applications. In this work, we propose Feedback-Driven Security Patching (FDSP), where LLMs automatically refine generated, vulnerable code. Our approach leverages automatic static code analysis to empower the LLM to generate and implement potential solutions to address vulnerabilities. We address the research communitys needs for safe code generation by introducing a large-scale dataset, PythonSecurityEval, covering the diversity of real-world applications, including databases, websites and operating systems. We empirically validate that FDSP outperforms prior work that uses self-feedback from LLMs by up to 17.6% through our procedure that injects targeted, external feedback. Code and data are available at \url{https://github.com/Kamel773/LLM-code-refine}