Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation
Atharva Mangeshkumar Agrawal, Rutika Pandurang Shinde, Vasanth Kumar, Bhukya, Ashmita Chakraborty, Sagar Bharat Shah, Tanmay Shukla, Sree Pradeep, Kumar Relangi, Nilesh Mutyam

TL;DR
This paper compares fine-tuning with LoRA and Retrieval-Augmented Generation techniques on large language models for doctor-patient dialogues, evaluating their performance in medical accuracy, coherence, and safety across multiple datasets.
Contribution
It provides a comprehensive analysis of two prominent LLM adaptation methods in healthcare dialogue, highlighting their respective strengths and limitations.
Findings
Retrieval-Augmented Generation improves factual accuracy.
Fine-tuning with LoRA enhances coherence and safety.
Model robustness varies across medical domains.
Abstract
Large language models (LLMs) have shown impressive capabilities in natural language processing tasks, including dialogue generation. This research aims to conduct a novel comparative analysis of two prominent techniques, fine-tuning with LoRA (Low-Rank Adaptation) and the Retrieval-Augmented Generation (RAG) framework, in the context of doctor-patient chat conversations with multiple datasets of mixed medical domains. The analysis involves three state-of-the-art models: Llama-2, GPT, and the LSTM model. Employing real-world doctor-patient dialogues, we comprehensively evaluate the performance of models, assessing key metrics such as language quality (perplexity, BLEU score), factual accuracy (fact-checking against medical knowledge bases), adherence to medical guidelines, and overall human judgments (coherence, empathy, safety). The findings provide insights into the strengths and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Cosine Annealing · Linear Warmup With Cosine Annealing · Tanh Activation · Discriminative Fine-Tuning · Attention Dropout · Layer Normalization · Linear Layer · Byte Pair Encoding
