Learned or Memorized ? Quantifying Memorization Advantage in Code LLMs

TL;DR

This paper introduces a perturbation-based method to measure memorization in code LLMs, revealing that memorization varies by model, task, and dataset, with implications for evaluation and security.

Contribution

It proposes a novel approach to quantify memorization advantage in code LLMs and evaluates multiple models and benchmarks to understand their generalization and memorization behaviors.

Findings

Memorization advantage varies widely across models and tasks.

Code summarization shows low sensitivity, test generation shows high sensitivity.

Certain benchmarks suspected of leakage actually show low memorization advantage.

Abstract

The lack of transparency about code datasets used to train large language models (LLMs) makes it difficult to detect, evaluate, and mitigate data leakage. We present a perturbation-based method to quantify memorization advantage in code LLMs, defined as the performance gap between likely seen and unseen inputs. We evaluate 8 open-source code LLMs on 19 benchmarks across four task families: code generation, code understanding, vulnerability detection, and bug fixing. Sensitivity patterns vary widely across models and tasks. For example, StarCoder reaches high sensitivity on some benchmarks (up to 0.8), while QwenCoder remains lower (mostly below 0.4), suggesting differences in generalization behavior. Task categories also differ: code summarization tends to show low sensitivity, whereas test generation is substantially higher. We then analyze two widely discussed benchmarks, CVEFixes and Defects4J, often suspected of leakage. Contrary to common concerns, both show low memorization advantage across models: CVEFixes remains below 0.1, and Defects4J is lower than other program repair benchmarks. These results suggest that, for these datasets, models may rely more on learned generalization than direct memorization. Overall, our findings provide evidence that memorization risk is highly task- and model-dependent, and highlight the need for stronger evaluation protocols, especially in security-focused settings.