The Competence Shadow: Theory and Bounds of AI Assistance in Safety Engineering

TL;DR

This paper develops a formal framework to analyze AI assistance in safety engineering, revealing how AI can both aid and hinder safety analysis depending on collaboration design.

Contribution

It introduces the competence shadow concept and formalizes performance bounds, emphasizing workflow design over tool qualification for trustworthy AI assistance.

Findings

AI assistance can cause systematic narrowing of human reasoning.

Performance degradation can exceed naive estimates due to the competence shadow.

Proper workflow design can prevent analysis degradation.

Abstract

As AI assistants become integrated into safety engineering workflows for Physical AI systems, a critical question emerges: does AI assistance improve safety analysis quality, or introduce systematic blind spots that surface only through post-deployment incidents? This paper develops a formal framework for AI assistance in safety analysis. We first establish why safety engineering resists benchmark-driven evaluation: safety competence is irreducibly multidimensional, constrained by context-dependent correctness, inherent incompleteness, and legitimate expert disagreement. We formalize this through a five-dimensional competence framework capturing domain knowledge, standards expertise, operational experience, contextual understanding, and judgment. We introduce the competence shadow: the systematic narrowing of human reasoning induced by AI-generated safety analysis. The shadow is not what the AI presents, but what it prevents from being considered. We formalize four canonical human-AI collaboration structures and derive closed-form performance bounds, demonstrating that the competence shadow compounds multiplicatively to produce degradation far exceeding naive additive estimates. The central finding is that AI assistance in safety engineering is a collaboration design problem, not a software procurement decision. The same tool degrades or improves analysis quality depending entirely on how it is used. We derive non-degradation conditions for shadow-resistant workflows and call for a shift from tool qualification toward workflow qualification for trustworthy Physical AI.