On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code

TL;DR

This paper evaluates the robustness of instruction-tuned large language models for coding, revealing significant vulnerabilities to input perturbations and proposing a mitigation strategy to improve their resilience.

Contribution

Introduces DegradePrompter, a systematic evaluation method for assessing robustness of Code LLMs against input challenges, and compares open-source and commercial models.

Findings

Open-source models show 12-34% decline in correctness under perturbations.

Commercial models are more resilient, with 3-24% performance degradation.

A simple mitigation strategy can enhance model robustness.

Abstract

The advent of instruction-tuned Large Language Models designed for coding tasks (Code LLMs) has transformed software engineering practices. However, their robustness against various input challenges remains a critical concern. This study introduces DegradePrompter, a novel method designed to systematically evaluate the robustness of instruction-tuned Code LLMs. We assess the impact of diverse input challenges on the functionality and correctness of generated code using rigorous metrics and established benchmarks. Our comprehensive evaluation includes five state-of-the-art open-source models and three production-grade closed-source models, revealing varying degrees of robustness. Open-source models demonstrate an increased susceptibility to input perturbations, resulting in declines in functional correctness ranging from 12% to 34%. In contrast, commercial models demonstrate relatively greater resilience, with performance degradation ranging from 3% to 24%. To enhance the robustness of the models against these vulnerabilities, we investigate a straightforward yet effective mitigation strategy. Our findings highlight the need for robust defense mechanisms and comprehensive evaluations during both the development and deployment phases to ensure the resilience and reliability of automated code generation systems.