Generative AI in Systems Engineering: A Framework for Risk Assessment of Large Language Models

TL;DR

This paper presents the LLM Risk Assessment Framework (LRF), a structured method to evaluate and manage risks of Large Language Models in systems engineering, promoting safe and reliable AI integration.

Contribution

It introduces a novel classification framework for LLM applications based on autonomy and impact, aiding risk assessment and validation in systems engineering.

Findings

Enables consistent risk level determination across development stages

Supports tailored validation and oversight strategies

Facilitates safe deployment of LLMs in engineering processes

Abstract

The increasing use of Large Language Models (LLMs) offers significant opportunities across the engineering lifecycle, including requirements engineering, software development, process optimization, and decision support. Despite this potential, organizations face substantial challenges in assessing the risks associated with LLM use, resulting in inconsistent integration, unknown failure modes, and limited scalability. This paper introduces the LLM Risk Assessment Framework (LRF), a structured approach for evaluating the application of LLMs within Systems Engineering (SE) environments. The framework classifies LLM-based applications along two fundamental dimensions: autonomy, ranging from supportive assistance to fully automated decision making, and impact, reflecting the potential severity of incorrect or misleading model outputs on engineering processes and system elements. By combining these dimensions, the LRF enables consistent determination of corresponding risk levels across the development lifecycle. The resulting classification supports organizations in identifying appropriate validation strategies, levels of human oversight, and required countermeasures to ensure safe and transparent deployment. The framework thereby helps align the rapid evolution of AI technologies with established engineering principles of reliability, traceability, and controlled process integration. Overall, the LRF provides a basis for risk-aware adoption of LLMs in complex engineering environments and represents a first step toward standardized AI assurance practices in systems engineering.