SafeLLM: Domain-Specific Safety Monitoring for Large Language Models: A Case Study of Offshore Wind Maintenance

TL;DR

This paper presents SafeLLM, a domain-specific safety monitoring system for large language models, applied to offshore wind maintenance, aiming to improve alarm interpretation and safety in operational decision-making.

Contribution

It introduces a specialized conversational agent that uses statistical sentence distance techniques to detect hallucinations and unsafe outputs in LLMs for offshore wind maintenance.

Findings

Preliminary testing with ChatGPT-4 shows potential for safe alarm interpretation.

The approach can be enhanced with re-training on specialized datasets.

Limitations identified in using ChatGPT-4 for safety monitoring.

Abstract

The Offshore Wind (OSW) industry is experiencing significant expansion, resulting in increased Operations \& Maintenance (O\&M) costs. Intelligent alarm systems offer the prospect of swift detection of component failures and process anomalies, enabling timely and precise interventions that could yield reductions in resource expenditure, as well as scheduled and unscheduled downtime. This paper introduces an innovative approach to tackle this challenge by capitalising on Large Language Models (LLMs). We present a specialised conversational agent that incorporates statistical techniques to calculate distances between sentences for the detection and filtering of hallucinations and unsafe output. This potentially enables improved interpretation of alarm sequences and the generation of safer repair action recommendations by the agent. Preliminary findings are presented with the approach applied to ChatGPT-4 generated test sentences. The limitation of using ChatGPT-4 and the potential for enhancement of this agent through re-training with specialised OSW datasets are discussed.