Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation

TL;DR

Sim2Val is a framework that uses paired simulation and real-world data to reduce variance in performance estimates, significantly decreasing the need for costly real-world testing in robotics.

Contribution

It introduces a novel variance reduction method using control variates from paired data, with theoretical guarantees and empirical validation in robotics.

Findings

Reduces variance of estimates by leveraging simulation data.

Achieves high-confidence bounds with fewer real-world samples.

Demonstrates improved sample efficiency in autonomous driving and robotics.

Abstract

Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive, and if conducted may still yield insufficient data for high-confidence guarantees. In this work we introduce Sim2Val, a general estimation framework that leverages paired data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly improved sample efficiency. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems.