Instance Performance Difference: A Metric to Measure the Sim-To-Real Gap in Camera Simulation

TL;DR

The paper introduces Instance Performance Difference (IPD), a metric to quantify the performance gap between synthetic and real images in robotics perception tasks, aiding in better synthetic data generation for real-world applications.

Contribution

It proposes the IPD metric to evaluate and improve the realism of synthetic images for perception tasks, enhancing sim-to-real transfer in robotics.

Findings

IPD effectively measures the realism gap in perception tasks.

IPD helps identify the most realistic synthetic image synthesis methods.

The metric supports creating synthetic datasets that match real-world perception performance.

Abstract

In this contribution, we introduce the concept of Instance Performance Difference (IPD), a metric designed to measure the gap in performance that a robotics perception task experiences when working with real vs. synthetic pictures. By pairing synthetic and real instances in the pictures and evaluating their performance similarity using perception algorithms, IPD provides a targeted metric that closely aligns with the needs of real-world applications. We explain and demonstrate this metric through a rock detection task in lunar terrain images, highlighting the IPD's effectiveness in identifying the most realistic image synthesis method. The metric is thus instrumental in creating synthetic image datasets that perform in perception tasks like real-world photo counterparts. In turn, this supports robust sim-to-real transfer for perception algorithms in real-world robotics applications.