Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis

TL;DR

This paper introduces a framework combining label-to-image synthesis and transferability measures to validate simulation-based testing of machine learning models, effectively addressing domain shift issues.

Contribution

It presents a novel approach that enables transferability analysis and validation of simulation-based testing for semantic segmentation, extendable to other classification tasks.

Findings

Strong correlation (0.7) between synthetic and real test results for cars and pedestrians.

Discriminator can distinguish real from synthetic data, but IoU correlation remains high.

Framework allows extensive testing with controlled simulations.

Abstract

Many machine learning applications can benefit from simulated data for systematic validation - in particular if real-life data is difficult to obtain or annotate. However, since simulations are prone to domain shift w.r.t. real-life data, it is crucial to verify the transferability of the obtained results. We propose a novel framework consisting of a generative label-to-image synthesis model together with different transferability measures to inspect to what extent we can transfer testing results of semantic segmentation models from synthetic data to equivalent real-life data. With slight modifications, our approach is extendable to, e.g., general multi-class classification tasks. Grounded on the transferability analysis, our approach additionally allows for extensive testing by incorporating controlled simulations. We validate our approach empirically on a semantic segmentation task on driving scenes. Transferability is tested using correlation analysis of IoU and a learned discriminator. Although the latter can distinguish between real-life and synthetic tests, in the former we observe surprisingly strong correlations of 0.7 for both cars and pedestrians.