RAG based Question-Answering for Contextual Response Prediction System

TL;DR

This paper presents an end-to-end RAG-based question-answering system tailored for customer service, demonstrating improved accuracy and relevance over existing methods through automated and human evaluations.

Contribution

It introduces a practical RAG framework for industry use, addressing data, evaluation, and cost challenges in deploying LLMs for customer support.

Findings

Outperforms BERT-based algorithms in accuracy and relevance

Automated and human evaluations validate effectiveness

Supports customer service agents by reducing workload

Abstract

Large Language Models (LLMs) have shown versatility in various Natural Language Processing (NLP) tasks, including their potential as effective question-answering systems. However, to provide precise and relevant information in response to specific customer queries in industry settings, LLMs require access to a comprehensive knowledge base to avoid hallucinations. Retrieval Augmented Generation (RAG) emerges as a promising technique to address this challenge. Yet, developing an accurate question-answering framework for real-world applications using RAG entails several challenges: 1) data availability issues, 2) evaluating the quality of generated content, and 3) the costly nature of human evaluation. In this paper, we introduce an end-to-end framework that employs LLMs with RAG capabilities for industry use cases. Given a customer query, the proposed system retrieves relevant knowledge documents and leverages them, along with previous chat history, to generate response suggestions for customer service agents in the contact centers of a major retail company. Through comprehensive automated and human evaluations, we show that this solution outperforms the current BERT-based algorithms in accuracy and relevance. Our findings suggest that RAG-based LLMs can be an excellent support to human customer service representatives by lightening their workload.