Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development

TL;DR

This paper demonstrates how safety engineering frameworks like FMEA and STPA can be applied to text-to-image ML development to identify social and ethical risks across different stages of the product pipeline.

Contribution

It introduces the novel application of safety frameworks to social and ethical risk analysis in ML, beyond traditional technical failure modes.

Findings

Safety frameworks uncover social and ethical hazards in ML workflows.

Analysis reveals hazards between models, users, and development teams.

Frameworks are valuable even with limited model information.

Abstract

Identifying potential social and ethical risks in emerging machine learning (ML) models and their applications remains challenging. In this work, we applied two well-established safety engineering frameworks (FMEA, STPA) to a case study involving text-to-image models at three stages of the ML product development pipeline: data processing, integration of a T2I model with other models, and use. Results of our analysis demonstrate the safety frameworks - both of which are not designed explicitly examine social and ethical risks - can uncover failure and hazards that pose social and ethical risks. We discovered a broad range of failures and hazards (i.e., functional, social, and ethical) by analyzing interactions (i.e., between different ML models in the product, between the ML product and user, and between development teams) and processes (i.e., preparation of training data or workflows for using an ML service/product). Our findings underscore the value and importance of examining beyond an ML model in examining social and ethical risks, especially when we have minimal information about an ML model.