SecureReviewer: Enhancing Large Language Models for Secure Code Review through Secure-aware Fine-tuning

TL;DR

SecureReviewer enhances large language models for secure code review by creating a specialized dataset, applying secure-aware fine-tuning, and introducing new evaluation metrics, significantly improving security issue detection and review comment quality.

Contribution

The paper introduces SecureReviewer, a novel approach that combines secure-aware fine-tuning, domain grounding, and a new evaluation metric to improve security-focused code review with LLMs.

Findings

SecureReviewer outperforms existing methods in security issue detection accuracy.

The approach improves the relevance and utility of generated review comments.

SecureBLEU effectively evaluates security-specific review comment quality.

Abstract

Identifying and addressing security issues during the early phase of the development lifecycle is critical for mitigating the long-term negative impacts on software systems. Code review serves as an effective practice that enables developers to check their teammates' code before integration into the codebase. To streamline the generation of review comments, various automated code review approaches have been proposed, where LLM-based methods have significantly advanced the capabilities of automated review generation. However, existing models primarily focus on general-purpose code review, their effectiveness in identifying and addressing security-related issues remains underexplored. Moreover, adapting existing code review approaches to target security issues faces substantial challenges, including data scarcity and inadequate evaluation metrics. To address these limitations, we propose SecureReviewer, a new approach designed for enhancing LLMs' ability to identify and resolve security-related issues during code review. Specifically, we first construct a dataset tailored for training and evaluating secure code review capabilities. Leveraging this dataset, we fine-tune LLMs to generate code review comments that can effectively identify security issues and provide fix suggestions with our proposed secure-aware fine-tuning strategy. To mitigate hallucination in LLMs and enhance the reliability of their outputs, we integrate the RAG technique, which grounds the generated comments in domain-specific security knowledge. Additionally, we introduce SecureBLEU, a new evaluation metric designed to assess the effectiveness of review comments in addressing security issues. Experimental results demonstrate that SecureReviewer outperforms state-of-the-art baselines in both security issue detection accuracy and the overall quality and practical utility of generated review comments.