SimuScene: Training and Benchmarking Code Generation to Simulate Physical Scenarios

TL;DR

SimuScene systematically trains and benchmarks large language models on simulating physical scenarios through code, revealing current limitations and proposing reinforcement learning methods to improve physical reasoning and code generation.

Contribution

First comprehensive dataset and evaluation framework for LLMs simulating physical scenarios, along with a reinforcement learning pipeline to enhance physical reasoning in code generation.

Findings

Strongest model achieves only 21.5% pass rate on physical simulation tasks.

Training with SimuScene data improves physical simulation accuracy.

Reinforcement learning with visual rewards enhances code generation performance.

Abstract

Large language models (LLMs) have been extensively studied for tasks like math competitions, complex coding, and scientific reasoning, yet their ability to accurately represent and simulate physical scenarios via code remains underexplored. We propose SimuScene, the first systematic study that trains and evaluates LLMs on simulating physical scenarios across five physics domains and 52 physical concepts. We build an automatic pipeline to collect data, with human verification to ensure quality. The final dataset contains 7,659 physical scenarios with 334 human-verified examples as the test set. We evaluated 10 contemporary LLMs and found that even the strongest model achieves only a 21.5% pass rate, demonstrating the difficulty of the task. Finally, we introduce a reinforcement learning pipeline with visual rewards that uses a vision-language model as a judge to train textual models. Experiments show that training with our data improves physical simulation via code while substantially enhancing general code generation performance.