ReGen: Generative Robot Simulation via Inverse Design
Phat Nguyen, Tsun-Hsuan Wang, Zhang-Wei Hong, Erfan Aasi, Andrew Silva, Guy Rosman, Sertac Karaman, Daniela Rus
TL;DR
ReGen is a novel framework that automates the creation of diverse and complex robot simulation environments from textual descriptions and behaviors using large language models, improving scalability and robustness in robot learning.
Contribution
It introduces a generative simulation method that uses inverse design and language models to automate and control the creation of robot simulation scenarios.
Findings
Generated environments are more diverse and complex than existing simulations.
High success rates in generating relevant simulation scenarios.
Enables controllable and counterfactual scenario generation.
Abstract
Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains a labor-intensive process. This paper introduces ReGen, a generative simulation framework that automates simulation design via inverse design. Given a robot's behavior -- such as a motion trajectory or an objective function -- and its textual description, ReGen infers plausible scenarios and environments that could have caused the behavior. ReGen leverages large language models to synthesize scenarios by expanding a directed graph that encodes cause-and-effect relationships, relevant entities, and their properties. This structured graph is then translated into a symbolic program, which configures and executes a robot simulation environment. Our framework supports (i) augmenting simulations based on ego-agent behaviors, (ii) controllable, counterfactual scenario generation,…
Peer Reviews
Decision·ICLR 2025 Poster
1. ReGen presents an inverse design approach for generative simulation, which allows for the creation of diverse and complex simulated environments based on agent behavior and textual descriptions. 2. ReGen generates more diverse and complex simulated environments compared to existing simulations, as demonstrated in autonomous driving and robot manipulation tasks. 3. ReGen enables controllable generation for corner cases, which is important for safety-critical applications like autonomous drivin
1. The evaluation of ReGen is primarily focused on diversity and complexity, with less emphasis on the realism and physical accuracy of the generated simulations. 2. ReGen heavily relies on LLMs, which can be computationally expensive and may not always generate semantically accurate or physically plausible scenarios due to LLMs. 3. The applicability of ReGen to other robotics domains beyond autonomous driving and robot manipulation is not extensively explored.
Diversifying and augmenting existing simulated environments and trajectories through counterfactural generation is an important research problem. It provides a promosing way to improve the robustness and safety of agents and policies under corner cases and unexpected scenarios.
The paper suffers from significant weaknesses in writing clarity and the depth of analysis, specifically: - The methodology section 2 does not detail the models, the model versions, and the important hyperparameters used for prompting LLM / VLM for counterfactual generation. Some of the details are deferred to page 9, which should have been in much earlier in the paper. - No prompt examples for LLM / VLM are provided in the appendix, making the method unreproducible. No concrete, detailed exampl
While I do not see a clear strength in terms of novelty. However, the authors provide a comprehensive evaluation against multiple baselines, and thorough implementation across autonomous driving and manipulation tasks to demonstrate the applicability of the proposed method.
The main contribution of this paper is vague. The use of LLMs to generate simulated task environments for robotic tasks cannot be considered as a major contribution. The proposed method of using LLMs for graph expansion and simulation generation appears to be a straightforward extension of existing work such as [a, b]. I do not see a convincing argument that states the technical contributions of the proposed approach other than some prompt engineering. The "inverse design" is essentially a rebra
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAutonomous Vehicle Technology and Safety · Reinforcement Learning in Robotics · Artificial Intelligence in Games