Multi-LLM Collaboration + Data-Centric Innovation = 2x Better Vulnerability Repair

TL;DR

This paper introduces VulMaster, a novel Transformer-based model that leverages multi-LLM collaboration and data-centric strategies to significantly improve automatic vulnerability repair in software, especially for complex code and expert knowledge integration.

Contribution

VulMaster combines multiple data types and LLM collaboration to enhance vulnerability repair, addressing limitations of existing DL-based methods in handling code structure and expert insights.

Findings

VulMaster outperforms state-of-the-art approaches in EM, BLEU, and CodeBLEU scores.

The model effectively handles lengthy code and incorporates expert knowledge.

Experimental results show substantial improvements on a large real-world dataset.

Abstract

The advances of deep learning (DL) have paved the way for automatic software vulnerability repair approaches, which effectively learn the mapping from the vulnerable code to the fixed code. Nevertheless, existing DL-based vulnerability repair methods face notable limitations: 1) they struggle to handle lengthy vulnerable code, 2) they treat code as natural language texts, neglecting its inherent structure, and 3) they do not tap into the valuable expert knowledge present in the expert system. To address this, we propose VulMaster, a Transformer-based neural network model that excels at generating vulnerability repairs through data-centric innovation. Specifically, VulMaster introduces the utilization and combination of various types of input data, including complete vulnerable code of any size, vulnerable code structures, and expert knowledge from the CWE system. Additionally, VulMaster leverages the collaboration between two Large Language Models (LLMs), CodeT5 and ChatGPT: CodeT5 acts as the customizable backbone LLM, fine-tuned with the training data, while ChatGPT supplements by providing missing relevant inputs to CodeT5. We evaluated VulMaster on a real-world C/C++ vulnerability repair dataset comprising 1,754 projects with 5,800 vulnerable functions. The experimental results demonstrated that VulMaster exhibits substantial improvements compared to the learning-based state-of-the-art vulnerability repair approach. Specifically, VulMaster improves the EM, BLEU, and CodeBLEU scores from 10.2\% to 20.0\%, 21.3\% to 29.3\%, and 32.5\% to 40.9\%, respectively.