MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering

TL;DR

MEnvAgent is a scalable multi-language framework that automates the construction of verifiable software environments, significantly improving success rates and efficiency for large language model-based software engineering tasks.

Contribution

We introduce MEnvAgent, a novel multi-agent framework with environment reuse mechanisms for scalable, verifiable environment construction across multiple programming languages.

Findings

Outperforms baselines with 8.6% higher Fail-to-Pass rate.

Reduces environment construction time by 43%.

Creates the largest open-source polyglot verifiable environment dataset.

Abstract

The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at https://github.com/ernie-research/MEnvAgent.