RE for AI in Practice: Managing Data Annotation Requirements for AI Autonomous Driving Systems

TL;DR

This paper investigates how data annotation requirements impact AI perception systems in autonomous driving, identifying challenges and best practices through interviews, and offers empirically grounded guidance for improving annotation quality and system reliability.

Contribution

It provides the first empirical analysis of annotation requirement challenges and best practices in autonomous driving AI, bridging RE and AI fields with actionable insights.

Findings

Identified five key challenges in annotation requirements.

Proposed three categories of best practices.

Uncovered relationships between requirements quality and AI system performance.

Abstract

High-quality data annotation requirements are crucial for the development of safe and reliable AI-enabled perception systems (AIePS) in autonomous driving. Although these requirements play a vital role in reducing bias and enhancing performance, their formulation and management remain underexplored, leading to inconsistencies, safety risks, and regulatory concerns. Our study investigates how annotation requirements are defined and used in practice, the challenges in ensuring their quality, practitioner-recommended improvements, and their impact on AIePS development and performance. We conducted $19$ semi-structured interviews with participants from six international companies and four research organisations. Our thematic analysis reveals five main key challenges: ambiguity, edge case complexity, evolving requirements, inconsistencies, and resource constraints and three main categories of best practices, including ensuring compliance with ethical standards, improving data annotation requirements guidelines, and embedded quality assurance for data annotation requirements. We also uncover critical interrelationships between annotation requirements, annotation practices, annotated data quality, and AIePS performance and development, showing how requirement flaws propagate through the AIePS development pipeline. To the best of our knowledge, this study is the first to offer empirically grounded guidance on improving annotation requirements, offering actionable insights to enhance annotation quality, regulatory compliance, and system reliability. It also contributes to the emerging fields of Software Engineering (SE for AI) and Requirements Engineering (RE for AI) by bridging the gap between RE and AI in a timely and much-needed manner.