Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning

TL;DR

This paper investigates the use of context-aware prompt tuning with large language models, specifically GitHub Copilot, to improve the detection and repair of buffer overflow vulnerabilities in code.

Contribution

It introduces a novel prompt tuning technique that injects domain knowledge, significantly enhancing Copilot's ability to repair buffer overflow vulnerabilities.

Findings

Vulnerability detection rate is 76% for buffer overflows.

Repair success rate is only 15% without domain knowledge.

Prompt tuning increases repair success to 63%, over four times improvement.

Abstract

Large Language Models (LLMs) have shown significant challenges in detecting and repairing vulnerable code, particularly when dealing with vulnerabilities involving multiple aspects, such as variables, code flows, and code structures. In this study, we utilize GitHub Copilot as the LLM and focus on buffer overflow vulnerabilities. Our experiments reveal a notable gap in Copilot's abilities when dealing with buffer overflow vulnerabilities, with a 76% vulnerability detection rate but only a 15% vulnerability repair rate. To address this issue, we propose context-aware prompt tuning techniques designed to enhance LLM performance in repairing buffer overflow. By injecting a sequence of domain knowledge about the vulnerability, including various security and code contexts, we demonstrate that Copilot's successful repair rate increases to 63%, representing more than four times the improvement compared to repairs without domain knowledge.