FDI: Attack Neural Code Generation Systems through User Feedback Channel

TL;DR

This paper reveals that user feedback channels in neural code generation systems are vulnerable to injection attacks, which can manipulate generated code to include vulnerabilities or malicious content, highlighting security risks.

Contribution

The paper systematically analyzes feedback mechanisms in neural code systems, demonstrating how they can be exploited through FDI attacks to inject malicious prompts or backdoors.

Findings

Feedback mechanisms are vulnerable to injection attacks.

FDI attacks can manipulate code generation to include vulnerabilities.

Stealthy attacks can introduce malicious code or spam.

Abstract

Neural code generation systems have recently attracted increasing attention to improve developer productivity and speed up software development. Typically, these systems maintain a pre-trained neural model and make it available to general users as a service (e.g., through remote APIs) and incorporate a feedback mechanism to extensively collect and utilize the users' reaction to the generated code, i.e., user feedback. However, the security implications of such feedback have not yet been explored. With a systematic study of current feedback mechanisms, we find that feedback makes these systems vulnerable to feedback data injection (FDI) attacks. We discuss the methodology of FDI attacks and present a pre-attack profiling strategy to infer the attack constraints of a targeted system in the black-box setting. We demonstrate two proof-of-concept examples utilizing the FDI attack surface to implement prompt injection attacks and backdoor attacks on practical neural code generation systems. The attacker may stealthily manipulate a neural code generation system to generate code with vulnerabilities, attack payload, and malicious and spam messages. Our findings reveal the security implications of feedback mechanisms in neural code generation systems, paving the way for increasing their security.