Identifying Systematic Errors in Object Detectors with the SCROD Pipeline

TL;DR

This paper presents a scalable, automated pipeline that combines generative models and physical simulators to generate synthetic street scenes with fine-grained control, aiming to identify systematic errors in object detectors for safety-critical applications.

Contribution

The authors introduce a novel hybrid framework that integrates the strengths of physical simulators and generative models for controlled synthetic data generation, along with an evaluation benchmark.

Findings

Effective generation of diverse street scenes with fine-grained control.

Identification of systematic errors in object detectors.

Provision of a standardized evaluation benchmark.

Abstract

The identification and removal of systematic errors in object detectors can be a prerequisite for their deployment in safety-critical applications like automated driving and robotics. Such systematic errors can for instance occur under very specific object poses (location, scale, orientation), object colors/textures, and backgrounds. Real images alone are unlikely to cover all relevant combinations. We overcome this limitation by generating synthetic images with fine-granular control. While generating synthetic images with physical simulators and hand-designed 3D assets allows fine-grained control over generated images, this approach is resource-intensive and has limited scalability. In contrast, using generative models is more scalable but less reliable in terms of fine-grained control. In this paper, we propose a novel framework that combines the strengths of both approaches. Our meticulously designed pipeline along with custom models enables us to generate street scenes with fine-grained control in a fully automated and scalable manner. Moreover, our framework introduces an evaluation setting that can serve as a benchmark for similar pipelines. This evaluation setting will contribute to advancing the field and promoting standardized testing procedures.