From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems

TL;DR

This paper proposes a proactive safety engineering approach for ML systems by integrating hazard analysis with LLM support to automate and streamline traditional safety methods, aiming to prevent hazards early in development.

Contribution

It introduces a novel method combining LLMs with modified STPA for hazard analysis, reducing reliance on experts and making safety assessment more accessible during ML system development.

Findings

LLMs can assist in automating hazard analysis tasks.

Modified STPA with LLM support anticipates unanticipated issues.

The approach improves safety planning efficiency.

Abstract

Machine learning (ML) components are increasingly integrated into software products, yet their complexity and inherent uncertainty often lead to unintended and hazardous consequences, both for individuals and society at large. Despite these risks, practitioners seldom adopt proactive approaches to anticipate and mitigate hazards before they occur. Traditional safety engineering approaches, such as Failure Mode and Effects Analysis (FMEA) and System Theoretic Process Analysis (STPA), offer systematic frameworks for early risk identification but are rarely adopted. This position paper advocates for integrating hazard analysis into the development of any ML-powered software product and calls for greater support to make this process accessible to developers. By using large language models (LLMs) to partially automate a modified STPA process with human oversight at critical steps, we expect to address two key challenges: the heavy dependency on highly experienced safety engineering experts, and the time-consuming, labor-intensive nature of traditional hazard analysis, which often impedes its integration into real-world development workflows. We illustrate our approach with a running example, demonstrating that many seemingly unanticipated issues can, in fact, be anticipated.