Ambiguous Annotations: When is a Pedestrian not a Pedestrian?

TL;DR

This paper investigates the ambiguity in human-annotated datasets for autonomous driving, showing that removing ambiguous data improves pedestrian detection performance and emphasizing the importance of understanding dataset properties.

Contribution

It introduces a method to identify and exclude ambiguous annotations, enhancing model accuracy and efficiency in pedestrian detection tasks.

Findings

Excluding ambiguous data improves detection metrics like LAMR, precision, and F1 score.

Removing ambiguous instances reduces training time and annotation costs.

Understanding dataset properties is crucial for safe and representative data exclusion.

Abstract

Datasets labelled by human annotators are widely used in the training and testing of machine learning models. In recent years, researchers are increasingly paying attention to label quality. However, it is not always possible to objectively determine whether an assigned label is correct or not. The present work investigates this ambiguity in the annotation of autonomous driving datasets as an important dimension of data quality. Our experiments show that excluding highly ambiguous data from the training improves model performance of a state-of-the-art pedestrian detector in terms of LAMR, precision and F1 score, thereby saving training time and annotation costs. Furthermore, we demonstrate that, in order to safely remove ambiguous instances and ensure the retained representativeness of the training data, an understanding of the properties of the dataset and class under investigation is crucial.