Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback

TL;DR

This paper introduces a retrieval-augmented, multi-tool feedback workflow that significantly improves the security and correctness of code generated by large language models through iterative self-repair guided by static analysis and symbolic execution.

Contribution

It presents a novel multi-tool, retrieval-augmented repair system that enhances LLM-generated code security and correctness by integrating static analysis, security scanning, and symbolic execution.

Findings

Security vulnerabilities reduced by 96% in DeepSeek.

Critical security defect rate decreased from 58.55% to 22.19% in CodeLlama.

System demonstrates significant robustness improvements.

Abstract

Large Language Models (LLMs) can generate code but often introduce security vulnerabilities, logical inconsistencies, and compilation errors. Prior work demonstrates that LLMs benefit substantially from structured feedback, static analysis, retrieval augmentation, and execution-based refinement. We propose a retrieval-augmented, multi-tool repair workflow in which a single code-generating LLM iteratively refines its outputs using compiler diagnostics, CodeQL security scanning, and KLEE symbolic execution. A lightweight embedding model is used for semantic retrieval of previously successful repairs, providing security-focused examples that guide generation. Evaluated on a combined dataset of 3,242 programs generated by DeepSeek-Coder-1.3B and CodeLlama-7B, the system demonstrates significant improvements in robustness. For DeepSeek, security vulnerabilities were reduced by 96%. For the larger CodeLlama model, the critical security defect rate was decreased from 58.55% to 22.19%, highlighting the efficacy of tool-assisted self-repair even on "stubborn" models.