Explainability as a Compliance Requirement: What Regulated Industries Need from AI Tools for Design Artifact Generation

TL;DR

This paper investigates the importance of explainability in AI tools for design artifact generation within regulated industries, highlighting challenges, impacts, and strategies to enhance transparency and compliance in safety-critical requirements engineering workflows.

Contribution

It provides empirical insights into the explainability gap in AI-driven RE tools and proposes practical features to improve transparency and regulatory compliance.

Findings

Non-explainable AI outputs require manual validation

Lack of explainability reduces stakeholder trust

Key improvements include source tracing and justification support

Abstract

Artificial Intelligence (AI) tools for automating design artifact generation are increasingly used in Requirements Engineering (RE) to transform textual requirements into structured diagrams and models. While these AI tools, particularly those based on Natural Language Processing (NLP), promise to improve efficiency, their adoption remains limited in regulated industries where transparency and traceability are essential. In this paper, we investigate the explainability gap in AI-driven design artifact generation through semi-structured interviews with ten practitioners from safety-critical industries. We examine how current AI-based tools are integrated into workflows and the challenges arising from their lack of explainability. We also explore mitigation strategies, their impact on project outcomes, and features needed to improve usability. Our findings reveal that non-explainable AI outputs necessitate extensive manual validation, reduce stakeholder trust, struggle to handle domain-specific terminology, disrupt team collaboration, and introduce regulatory compliance risks, often negating the anticipated efficiency benefits. To address these issues, we identify key improvements, including source tracing, providing clear justifications for tool-generated decisions, supporting domain-specific adaptation, and enabling compliance validation. This study outlines a practical roadmap for improving the transparency, reliability, and applicability of AI tools in requirements engineering workflows, particularly in regulated and safety-critical environments where explainability is crucial for adoption and certification.