Let's Roll: Synthetic Dataset Analysis for Pedestrian Detection Across Different Shutter Types

TL;DR

This study uses synthetic data generated with Unreal Engine 5 to compare pedestrian detection accuracy between global shutter and rolling shutter cameras, revealing similar performance for coarse detection but differences in fine-grained accuracy.

Contribution

It introduces a synthetic dataset for analyzing the impact of shutter types on ML pedestrian detection, highlighting the need for further research on RS effects in ISP-less pipelines.

Findings

Detection performance is similar for coarse detection between GS and RS.

Significant differences exist in fine-grained detection accuracy.

RS effects may require correction in ISP-less ML pipelines for precise localization.

Abstract

Computer vision (CV) pipelines are typically evaluated on datasets processed by image signal processing (ISP) pipelines even though, for resource-constrained applications, an important research goal is to avoid as many ISP steps as possible. In particular, most CV datasets consist of global shutter (GS) images even though most cameras today use a rolling shutter (RS). This paper studies the impact of different shutter mechanisms on machine learning (ML) object detection models on a synthetic dataset that we generate using the advanced simulation capabilities of Unreal Engine 5 (UE5). In particular, we train and evaluate mainstream detection models with our synthetically-generated paired GS and RS datasets to ascertain whether there exists a significant difference in detection accuracy between these two shutter modalities, especially when capturing low-speed objects (e.g., pedestrians). The results of this emulation framework indicate the performance between them are remarkably congruent for coarse-grained detection (mean average precision (mAP) for IOU=0.5), but have significant differences for fine-grained measures of detection accuracy (mAP for IOU=0.5:0.95). This implies that ML pipelines might not need explicit correction for RS for many object detection applications, but mitigating RS effects in ISP-less ML pipelines that target fine-grained location of the objects may need additional research.