"Where is My Troubleshooting Procedure?": Studying the Potential of RAG in Assisting Failure Resolution of Large Cyber-Physical System

TL;DR

This paper investigates how Retrieval Augmented Generation (RAG) can support operators in quickly finding troubleshooting procedures in complex industrial manuals, enhancing failure response in cyber-physical systems.

Contribution

It provides experimental evidence on RAG's effectiveness in assisting failure resolution in large cyber-physical systems, highlighting practical considerations.

Findings

RAG can help operators retrieve troubleshooting procedures more efficiently.

Operators need to cross-validate RAG recommendations before acting.

RAG improves response times during failure incidents.

Abstract

In today's complex industrial environments, operators must often navigate through extensive technical manuals to identify troubleshooting procedures that may help react to some observed failure symptoms. These manuals, written in natural language, describe many steps in detail. Unfortunately, the number, magnitude, and articulation of these descriptions can significantly slow down and complicate the retrieval of the correct procedure during critical incidents. Interestingly, Retrieval Augmented Generation (RAG) enables the development of tools based on conversational interfaces that can assist operators in their retrieval tasks, improving their capability to respond to incidents. This paper presents the results of a set of experiments that derive from the analysis of the troubleshooting procedures available in Fincantieri, a large international company developing complex naval cyber-physical systems. Results show that RAG can assist operators in reacting promptly to failure symptoms, although specific measures have to be taken into consideration to cross-validate recommendations before actuating them.