System Safety Engineering for Social and Ethical ML Risks: A Case Study

TL;DR

This paper explores applying systems safety engineering, specifically STPA, to analyze and mitigate social and ethical risks in ML-driven Prescription Drug Monitoring Programs, aiming to improve risk management in complex sociotechnical systems.

Contribution

It demonstrates how systems safety analysis can be adapted to identify social and ethical risks in ML systems, providing concrete design controls for risk mitigation.

Findings

STPA can identify social and ethical risks in ML systems.

Application to PDMPs reveals specific risk control strategies.

Enhances traditional risk analysis with safety engineering tools.

Abstract

Governments, industry, and academia have undertaken efforts to identify and mitigate harms in ML-driven systems, with a particular focus on social and ethical risks of ML components in complex sociotechnical systems. However, existing approaches are largely disjointed, ad-hoc and of unknown effectiveness. Systems safety engineering is a well established discipline with a track record of identifying and managing risks in many complex sociotechnical domains. We adopt the natural hypothesis that tools from this domain could serve to enhance risk analyses of ML in its context of use. To test this hypothesis, we apply a "best of breed" systems safety analysis, Systems Theoretic Process Analysis (STPA), to a specific high-consequence system with an important ML-driven component, namely the Prescription Drug Monitoring Programs (PDMPs) operated by many US States, several of which rely on an ML-derived risk score. We focus in particular on how this analysis can extend to identifying social and ethical risks and developing concrete design-level controls to mitigate them.