Predictive Inequity in Object Detection

TL;DR

This paper examines whether state-of-the-art object detection systems perform equitably across pedestrians with different skin tones, revealing disparities not explained by scene difficulty or occlusion.

Contribution

It introduces a dataset with annotated Fitzpatrick skin tones and analyzes predictive bias in object detection, exploring factors influencing this disparity.

Findings

Disparity exists between skin tone groups in detection accuracy.

Time of day and occlusion do not fully explain the bias.

Reweighting training loss impacts predictive performance.

Abstract

In this work, we investigate whether state-of-the-art object detection systems have equitable predictive performance on pedestrians with different skin tones. This work is motivated by many recent examples of ML and vision systems displaying higher error rates for certain demographic groups than others. We annotate an existing large scale dataset which contains pedestrians, BDD100K, with Fitzpatrick skin tones in ranges [1-3] or [4-6]. We then provide an in-depth comparative analysis of performance between these two skin tone groupings, finding that neither time of day nor occlusion explain this behavior, suggesting this disparity is not merely the result of pedestrians in the 4-6 range appearing in more difficult scenes for detection. We investigate to what extent time of day, occlusion, and reweighting the supervised loss during training affect this predictive bias.