Dynamic Multi-Agent Orchestration and Retrieval for Multi-Source Question-Answer Systems using Large Language Models

TL;DR

This paper introduces a multi-agent, dynamic retrieval framework for multi-source question-answer systems using LLMs, integrating diverse data sources with adaptive strategies to improve accuracy and relevance.

Contribution

It presents a novel multi-agent orchestration and dynamic retrieval methodology that combines unstructured and structured data sources with real-time prompt adaptation.

Findings

Enhanced response accuracy in contract management domain

Effective integration of unstructured PDFs and structured databases

Scalable framework for multi-source QA systems

Abstract

We propose a methodology that combines several advanced techniques in Large Language Model (LLM) retrieval to support the development of robust, multi-source question-answer systems. This methodology is designed to integrate information from diverse data sources, including unstructured documents (PDFs) and structured databases, through a coordinated multi-agent orchestration and dynamic retrieval approach. Our methodology leverages specialized agents-such as SQL agents, Retrieval-Augmented Generation (RAG) agents, and router agents - that dynamically select the most appropriate retrieval strategy based on the nature of each query. To further improve accuracy and contextual relevance, we employ dynamic prompt engineering, which adapts in real time to query-specific contexts. The methodology's effectiveness is demonstrated within the domain of Contract Management, where complex queries often require seamless interaction between unstructured and structured data. Our results indicate that this approach enhances response accuracy and relevance, offering a versatile and scalable framework for developing question-answer systems that can operate across various domains and data sources.