MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistants

TL;DR

This paper introduces MaPPing Your Model, a prompt-based attack that subtly manipulates LLMs to introduce security vulnerabilities in generated code, demonstrating its effectiveness across various models and highlighting security risks.

Contribution

The paper presents the MaPP attack, a novel prompt manipulation technique that causes LLMs to add vulnerabilities, revealing security risks in LLM-based programming assistants.

Findings

MaPP prompts are effective across multiple LLMs.

Scaling models does not mitigate MaPP attack effectiveness.

MaPP can target specific vulnerabilities in generated code.

Abstract

LLM-based programming assistants offer the promise of programming faster but with the risk of introducing more security vulnerabilities. Prior work has studied how LLMs could be maliciously fine-tuned to suggest vulnerabilities more often. With the rise of agentic LLMs, which may use results from an untrusted third party, there is a growing risk of attacks on the model's prompt. We introduce the Malicious Programming Prompt (MaPP) attack, in which an attacker adds a small amount of text to a prompt for a programming task (under 500 bytes). We show that our prompt strategy can cause an LLM to add vulnerabilities while continuing to write otherwise correct code. We evaluate three prompts on seven common LLMs, from basic to state-of-the-art commercial models. Using the HumanEval benchmark, we find that our prompts are broadly effective, with no customization required for different LLMs. Furthermore, the LLMs that are best at HumanEval are also best at following our malicious instructions, suggesting that simply scaling language models will not prevent MaPP attacks. Using a dataset of eight CWEs in 16 scenarios, we find that MaPP attacks are also effective at implementing specific and targeted vulnerabilities across a range of models. Our work highlights the need to secure LLM prompts against manipulation as well as rigorously auditing code generated with the help of LLMs.