Data Annotation Quality Problems in AI-Enabled Perception System Development

TL;DR

This paper investigates the common annotation errors in AI perception systems for automated driving, providing a taxonomy and insights from industry case studies to improve data quality and system safety.

Contribution

It presents a comprehensive taxonomy of annotation errors across the supply chain, validated with industry practitioners, and offers a diagnostic framework for enhancing data quality in AI development.

Findings

Identified 18 recurring annotation error types across three data-quality dimensions.

Validated taxonomy as a useful tool for root-cause analysis and quality improvement.

Provided actionable guidance for building trustworthy AI perception systems.

Abstract

Data annotation is essential but highly error-prone in the development of AI-enabled perception systems (AIePS) for automated driving, and its quality directly influences model performance, safety, and reliability. However, the industry lacks empirical insights into how annotation errors emerge and spread across the multi-organisational automotive supply chain. This study addresses this gap through a multi-organisation case study involving six companies and four research institutes across Europe and the UK. Based on 19 semi-structured interviews with 20 experts (50 hours of transcripts) and a six-phase thematic analysis, we develop a taxonomy of 18 recurring annotation error types across three data-quality dimensions: completeness (e.g., attribute omission, missing feedback loops, edge-case omissions, selection bias), accuracy (e.g., mislabelling, bounding-box inaccuracies, granularity mismatches, bias-driven errors), and consistency (e.g., inter-annotator disagreement, ambiguous instructions, misaligned hand-offs, cross-modality inconsistencies). The taxonomy was validated with industry practitioners, who reported its usefulness for root-cause analysis, supplier quality reviews, onboarding, and improving annotation guidelines. They described it as a failure-mode catalogue similar to FMEA. By conceptualising annotation quality as a lifecycle and supply-chain issue, this study contributes to SE4AI by offering a shared vocabulary, diagnostic toolset, and actionable guidance for building trustworthy AI-enabled perception systems.