Validating Robotics Simulators on Real-World Impacts

TL;DR

This paper evaluates how well popular robotics simulators replicate real-world impact dynamics, highlighting their strengths and limitations in modeling impacts and contact physics for different scenarios.

Contribution

It systematically compares Drake, MuJoCo, and Bullet simulators against real data, identifying key modeling differences and their effects on simulation accuracy.

Findings

Simulators accurately model inelastic impacts

Simulators struggle with elastic impact accuracy

Model differences limit high-dimensional simulation fidelity

Abstract

A realistic simulation environment is an essential tool in every roboticist's toolkit, with uses ranging from planning and control to training policies with reinforcement learning. Despite the centrality of simulation in modern robotics, little work has been done to compare the performance of robotics simulators against real-world data, especially for scenarios involving dynamic motions with high speed impact events. Handling dynamic contact is the computational bottleneck for most simulations, and thus the modeling and algorithmic choices surrounding impacts and friction form the largest distinctions between popular tools. Here, we evaluate the ability of several simulators to reproduce real-world trajectories involving impacts. Using experimental data, we identify system-specific contact parameters of popular simulators Drake, MuJoCo, and Bullet, analyzing the effects of modeling choices around these parameters. For the simple example of a cube tossed onto a table, simulators capture inelastic impacts well while failing to capture elastic impacts. For the higher-dimensional case of a Cassie biped landing from a jump, the simulators capture the bulk motion well but the accuracy is limited by numerous model differences between the real robot and the simulators.