Reliability in Semantic Segmentation: Can We Use Synthetic Data?

TL;DR

This paper introduces a synthetic data generation method using fine-tuned Stable Diffusion to evaluate and improve the robustness and out-of-distribution detection of semantic segmentation models in safety-critical applications.

Contribution

It demonstrates how synthetic data can be used to assess real-world reliability of segmentation models and improve their OOD detection capabilities.

Findings

High correlation between synthetic OOD evaluation and real OOD performance

Synthetic data effectively reveals model robustness to out-of-distribution inputs

Approach enhances model calibration and OOD detection

Abstract

Assessing the robustness of perception models to covariate shifts and their ability to detect out-of-distribution (OOD) inputs is crucial for safety-critical applications such as autonomous vehicles. By nature of such applications, however, the relevant data is difficult to collect and annotate. In this paper, we show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. By fine-tuning Stable Diffusion with only in-domain data, we perform zero-shot generation of visual scenes in OOD domains or inpainted with OOD objects. This synthetic data is employed to evaluate the robustness of pretrained segmenters, thereby offering insights into their performance when confronted with real edge cases. Through extensive experiments, we demonstrate a high correlation between the performance of models when evaluated on our synthetic OOD data and when evaluated on real OOD inputs, showing the relevance of such virtual testing. Furthermore, we demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters. Code and data are made public.