A Data Annotation Requirements Representation and Specification (DARS)

TL;DR

This paper introduces DARS, a framework for representing and specifying data annotation requirements to improve annotation quality and reliability in AI-enabled cyber-physical systems.

Contribution

DARS provides a novel structured approach with tools like the Annotation Negotiation Card and Scenario-Based Specification for requirement clarity and stakeholder alignment.

Findings

DARS reduces annotation errors related to completeness, accuracy, and consistency.

Evaluation on automotive perception data demonstrates DARS's effectiveness.

Framework enhances requirement engineering for data-dependent AI systems.

Abstract

With the rise of AI-enabled cyber-physical systems, data annotation has become a critical yet often overlooked process in the development of these intelligent information systems. Existing work in requirements engineering (RE) has explored how requirements for AI systems and their data can be represented. However, related interviews with industry professionals show that data annotations and their related requirements introduce distinct challenges, indicating a need for annotation-specific requirement representations. We propose the Data Annotation Requirements Representation and Specification (DARS), including an Annotation Negotiation Card to align stakeholders on objectives and constraints, and a Scenario-Based Annotation Specification to express atomic and verifiable data annotation requirements. We evaluate DARS with an automotive perception case related to an ongoing project, and a mapping against 18 real-world data annotation error types. The results suggest that DARS mitigates root causes of completeness, accuracy, and consistency annotation errors. By integrating DARS into RE, this work improves the reliability of safety-critical systems using data annotations and demonstrates how engineering frameworks must evolve for data-dependent components of today's intelligent information systems.