Environment-Aware Code Generation: How far are We?

TL;DR

This paper systematically evaluates large language models' ability to generate environment-specific, executable code, introducing a new benchmark and adaptation strategies to improve real-world applicability.

Contribution

It presents the first comprehensive study of environment-aware code generation, along with a new benchmark and adaptation methods to enhance LLMs' environment compatibility.

Findings

Current LLMs struggle with environment-specific code generation.

Adaptation strategies improve code executability across environments.

VersiBCB captures complex, evolving software environments.

Abstract

Recent progress in large language models (LLMs) has improved code generation, but most evaluations still test isolated, small-scale code (e.g., a single function) under default or unspecified software environments. As a result, it is unclear whether LLMs can reliably generate executable code tailored to a user's specific environment. We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations. To enable realistic evaluation, we introduce VersiBCB, a benchmark that is multi-package, execution-verified, and deprecation-aware, capturing complex and evolving environments that prior datasets often overlook. Using VersiBCB, we investigate three complementary adaptation axes: data, parameters, and cache, and develop representative strategies for each. Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability. These findings highlight key challenges and opportunities for deploying LLMs in practical software engineering workflows.