Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions

TL;DR

This study empirically evaluates the capabilities and limitations of state-of-the-art multi-agent LLM systems, like GPT-4.1 and Claude Sonnet 4, in automating dataset adaptation tasks in software engineering, highlighting performance gaps and improvement strategies.

Contribution

First empirical analysis of multi-agent LLM systems applied to dataset adaptation in software engineering, identifying key challenges and potential enhancements.

Findings

Systems can identify key files and generate partial adaptations.

Prompt interventions improve structural similarity from 7.25% to 67.14%.

Current systems rarely produce functionally correct implementations.

Abstract

Automating the adaptation of software engineering (SE) research artifacts across datasets is essential for scalability and reproducibility, yet it remains largely unstudied. Recent advances in large language model (LLM)-based multi-agent systems, such as GitHub Copilot's agent mode, promise to automate complex development workflows through coordinated reasoning, code generation, and tool interaction. This paper presents the first empirical study on how state-of-the-art multi-agent systems perform in dataset adaptation tasks. We evaluate Copilot, backed by GPT-4.1 and Claude Sonnet 4, on adapting SE research artifacts from benchmark repositories including ROCODE and LogHub2.0. Through a five-stage evaluation pipeline (file comprehension, code editing, command generation, validation, and final execution), we measure success rates, analyze failure patterns, and assess prompt-based interventions designed to enhance agent performance. Results show that current systems can identify key files and generate partial adaptations but rarely produce functionally correct implementations. Prompt-level interventions, especially providing execution error messages and reference code, substantially improve structural similarity to ground truth (from 7.25% to 67.14%), highlighting the importance of contextual and feedback-driven guidance. Our findings reveal both the promise and limitations of today's multi-agent LLM systems for dataset adaptation, and suggest concrete directions for building more reliable, self-correcting agents in future SE research.