Retrieval-Augmented Multi-LLM Ensemble for Industrial Part Specification Extraction

TL;DR

This paper presents RAGsemble, a retrieval-augmented multi-LLM ensemble system that significantly improves industrial part specification extraction from unstructured text by combining multiple models and grounding outputs with factual data.

Contribution

Introduction of a novel multi-LLM ensemble framework with retrieval augmentation, enhancing accuracy and reliability in industrial text extraction tasks.

Findings

Significant accuracy improvements over single-LLM baselines.

Effective grounding of outputs using FAISS-based retrieval.

Robust system architecture suitable for industrial deployment.

Abstract

Industrial part specification extraction from unstructured text remains a persistent challenge in manufacturing, procurement, and maintenance, where manual processing is both time-consuming and error-prone. This paper introduces a retrieval-augmented multi-LLM ensemble framework that orchestrates nine state-of-the-art Large Language Models (LLMs) within a structured three-phase pipeline. RAGsemble addresses key limitations of single-model systems by combining the complementary strengths of model families including Gemini (2.0, 2.5, 1.5), OpenAI (GPT-4o, o4-mini), Mistral Large, and Gemma (1B, 4B, 3n-e4b), while grounding outputs in factual data using FAISS-based semantic retrieval. The system architecture consists of three stages: (1) parallel extraction by diverse LLMs, (2) targeted research augmentation leveraging high-performing models, and (3) intelligent synthesis with conflict resolution and confidence-aware scoring. RAG integration provides real-time access to structured part databases, enabling the system to validate, refine, and enrich outputs through similarity-based reference retrieval. Experimental results using real industrial datasets demonstrate significant gains in extraction accuracy, technical completeness, and structured output quality compared to leading single-LLM baselines. Key contributions include a scalable ensemble architecture for industrial domains, seamless RAG integration throughout the pipeline, comprehensive quality assessment mechanisms, and a production-ready solution suitable for deployment in knowledge-intensive manufacturing environments.