Cross-Format Retrieval-Augmented Generation in XR with LLMs for Context-Aware Maintenance Assistance

TL;DR

This paper evaluates a Retrieval-Augmented Generation system using large language models to improve maintenance support across various data formats, demonstrating significant performance gains with advanced models like GPT-4.

Contribution

It provides a comprehensive assessment of LLMs in RAG systems for multi-format data retrieval and instruction generation in maintenance contexts, highlighting performance differences.

Findings

GPT-4 outperforms other models in response accuracy

Advanced models excel in complex, multi-format queries

System delivers timely, accurate maintenance responses

Abstract

This paper presents a detailed evaluation of a Retrieval-Augmented Generation (RAG) system that integrates large language models (LLMs) to enhance information retrieval and instruction generation for maintenance personnel across diverse data formats. We assessed the performance of eight LLMs, emphasizing key metrics such as response speed and accuracy, which were quantified using BLEU and METEOR scores. Our findings reveal that advanced models like GPT-4 and GPT-4o-mini significantly outperform their counterparts, particularly when addressing complex queries requiring multi-format data integration. The results validate the system's ability to deliver timely and accurate responses, highlighting the potential of RAG frameworks to optimize maintenance operations. Future research will focus on refining retrieval techniques for these models and enhancing response generation, particularly for intricate scenarios, ultimately improving the system's practical applicability in dynamic real-world environments.