Retrieval Augmented Generation for Domain-specific Question Answering

TL;DR

This paper presents a retrieval-augmented framework for domain-specific question answering that improves accuracy and reduces hallucinations by fine-tuning a retriever with a large question-answer database, demonstrated on Adobe products.

Contribution

It introduces a novel retrieval-aware fine-tuning approach for large language models tailored to domain-specific QA tasks, enhancing factual accuracy.

Findings

Significant improvement in answer accuracy with retrieval-aware fine-tuning

Reduction in hallucinations during answer generation

Effective grounding of responses with up-to-date retrieval data

Abstract

Question answering (QA) has become an important application in the advanced development of large language models. General pre-trained large language models for question-answering are not trained to properly understand the knowledge or terminology for a specific domain, such as finance, healthcare, education, and customer service for a product. To better cater to domain-specific understanding, we build an in-house question-answering system for Adobe products. We propose a novel framework to compile a large question-answer database and develop the approach for retrieval-aware finetuning of a Large Language model. We showcase that fine-tuning the retriever leads to major improvements in the final generation. Our overall approach reduces hallucinations during generation while keeping in context the latest retrieval information for contextual grounding.