What You See Is Not Always What You Get: Evaluating GPT's Comprehension of Source Code

TL;DR

This paper investigates the vulnerability of large language models to imperceptible, character-level adversarial attacks on source code, revealing significant susceptibility and emphasizing the need for more robust models in software engineering tasks.

Contribution

The study introduces four categories of imperceptible character attacks and systematically evaluates their impact on state-of-the-art LLMs' code comprehension capabilities.

Findings

LLMs are vulnerable to character-level perturbations

Perturbations cause significant performance degradation

Robustness varies across different LLMs

Abstract

Recent studies have demonstrated outstanding capabilities of large language models (LLMs) in software engineering tasks, including code generation and comprehension. While LLMs have shown significant potential in assisting with coding, LLMs are vulnerable to adversarial attacks. In this paper, we investigate the vulnerability of LLMs to imperceptible attacks. This class of attacks manipulate source code at the character level, which renders the changes invisible to human reviewers yet effective in misleading LLMs' behaviour. We devise these attacks into four distinct categories and analyse their impacts on code analysis and comprehension tasks. These four types of imperceptible character attacks include coding reordering, invisible coding characters, code deletions, and code homoglyphs. To assess the robustness of state-of-the-art LLMs, we present a systematic evaluation across multiple models using both perturbed and clean code snippets. Two evaluation metrics, model confidence using log probabilities of response and response correctness, are introduced. The results reveal that LLMs are susceptible to imperceptible coding perturbations, with varying degrees of degradation highlighted across different LLMs. Furthermore, we observe a consistent negative correlation between perturbation magnitude and model performance. These results highlight the urgent need for robust LLMs capable of manoeuvring behaviours under imperceptible adversarial conditions.