Advancing Risk and Quality Assurance: A RAG Chatbot for Improved Regulatory Compliance

TL;DR

This paper introduces a Retrieval Augmented Generation (RAG) chatbot that improves risk and quality assurance in regulated industries by enhancing query processing and compliance, demonstrated through real-world evaluations and hyperparameter analysis.

Contribution

The paper presents a novel RAG system with hybrid search and relevance boosting tailored for R&Q queries, outperforming traditional approaches and providing practical insights for deployment.

Findings

Significant performance improvements over traditional RAG methods

Effective handling of complex regulatory queries in real-world settings

Comprehensive hyperparameter analysis informing system optimization

Abstract

Risk and Quality (R&Q) assurance in highly regulated industries requires constant navigation of complex regulatory frameworks, with employees handling numerous daily queries demanding accurate policy interpretation. Traditional methods relying on specialized experts create operational bottlenecks and limit scalability. We present a novel Retrieval Augmented Generation (RAG) system leveraging Large Language Models (LLMs), hybrid search and relevance boosting to enhance R&Q query processing. Evaluated on 124 expert-annotated real-world queries, our actively deployed system demonstrates substantial improvements over traditional RAG approaches. Additionally, we perform an extensive hyperparameter analysis to compare and evaluate multiple configuration setups, delivering valuable insights to practitioners.