Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

TL;DR

This paper introduces OBFUSEVAL, a code obfuscation benchmark to accurately evaluate large language models' code generation capabilities by testing them on unseen, obfuscated code, revealing their true generalization ability.

Contribution

The paper proposes a novel obfuscation-based evaluation benchmark for LLMs, addressing dataset exposure and timeliness issues to better assess true code generation capabilities.

Findings

Obfuscation reduces test pass rate by up to 62.5%.

Current datasets may overestimate LLM capabilities due to exposure.

Obfuscation strategies reveal LLMs' limitations in unseen code.

Abstract

Recently, large language models (LLMs) have shown strong potential in code generation tasks. However, there are still gaps before they can be fully applied in actual software development processes. Accurately assessing the code generation capabilities of large language models has become an important basis for evaluating and improving the models. Some existing works have constructed datasets to evaluate the capabilities of these models. However, the current evaluation process may encounter the illusion of "Specialist in Familiarity", primarily due to three gaps: the exposure of target code, case timeliness, and dependency availability. The fundamental reason for these gaps is that the code in current datasets may have been extensively exposed and exercised during the training phase, and due to the continuous training and development of LLM, their timeliness has been severely compromised. The key to solve the problem is to, as much as possible, evaluate the LLMs using code that they have not encountered before. Thus, the fundamental idea in this paper is to draw on the concept of code obfuscation, changing code at different levels while ensuring the functionality and output. To this end, we build a code-obfuscation based benchmark OBFUSEVAL. We first collect 1,354 raw cases from five real-world projects, including function description and code. Then we use three-level strategy (symbol, structure and semantic) to obfuscate descriptions, code and context dependencies. We evaluate four LLMs on OBFU- SEVAL and compared the effectiveness of different obfuscation strategy. We use official test suites of these projects to evaluate the generated code. The results show that after obfuscation, the average decrease ratio of test pass rate can up to 62.5%.