BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI

TL;DR

BlueCodeAgent is a comprehensive blue teaming framework that leverages automated red teaming to improve the detection of unsafe code generated by large language models, significantly enhancing safety measures.

Contribution

The paper introduces BlueCodeAgent, a novel end-to-end blue teaming system that integrates automated red teaming to improve detection of unsafe code scenarios in LLMs.

Findings

Achieves 12.7% average F1 score improvement across tasks.

Effectively reduces false positives in vulnerable code detection.

Demonstrates continuous improvement through red teaming feedback.

Abstract

As large language models (LLMs) are increasingly used for code generation, concerns over the security risks have grown substantially. Early research has primarily focused on red teaming, which aims to uncover and evaluate vulnerabilities and risks of CodeGen models. However, progress on the blue teaming side remains limited, as developing defense requires effective semantic understanding to differentiate the unsafe from the safe. To fill in this gap, we propose BlueCodeAgent, an end-to-end blue teaming agent enabled by automated red teaming. Our framework integrates both sides: red teaming generates diverse risky instances, while the blue teaming agent leverages these to detect previously seen and unseen risk scenarios through constitution and code analysis with agentic integration for multi-level defense. Our evaluation across three representative code-related tasks--bias instruction detection, malicious instruction detection, and vulnerable code detection--shows that BlueCodeAgent achieves significant gains over the base models and safety prompt-based defenses. In particular, for vulnerable code detection tasks, BlueCodeAgent integrates dynamic analysis to effectively reduce false positives, a challenging problem as base models tend to be over-conservative, misclassifying safe code as unsafe. Overall, BlueCodeAgent achieves an average 12.7\% F1 score improvement across four datasets in three tasks, attributed to its ability to summarize actionable constitutions that enhance context-aware risk detection. We demonstrate that the red teaming benefits the blue teaming by continuously identifying new vulnerabilities to enhance defense performance.