Software Engineering Agents for Embodied Controller Generation : A Study in Minigrid Environments

TL;DR

This paper evaluates Software Engineering Agents (SWE-Agents) in generating controllers for embodied tasks in Minigrid environments, analyzing how information access impacts their performance and establishing a new evaluation domain.

Contribution

First extended evaluation of SWE-Agents on embodied tasks, comparing static and dynamic information access, and establishing a new benchmark for future research.

Findings

Information access significantly affects SWE-Agent performance.

Dynamic exploration improves task-solving success.

Static code analysis alone is often insufficient.

Abstract

Software Engineering Agents (SWE-Agents) have proven effective for traditional software engineering tasks with accessible codebases, but their performance for embodied tasks requiring well-designed information discovery remains unexplored. We present the first extended evaluation of SWE-Agents on controller generation for embodied tasks, adapting Mini-SWE-Agent (MSWEA) to solve 20 diverse embodied tasks from the Minigrid environment. Our experiments compare agent performance across different information access conditions: with and without environment source code access, and with varying capabilities for interactive exploration. We quantify how different information access levels affect SWE-Agent performance for embodied tasks and analyze the relative importance of static code analysis versus dynamic exploration for task solving. This work establishes controller generation for embodied tasks as a crucial evaluation domain for SWE-Agents and provides baseline results for future research in efficient reasoning systems.