Comparative Analysis of Drug-GPT and ChatGPT LLMs for Healthcare Insights: Evaluating Accuracy and Relevance in Patient and HCP Contexts

TL;DR

This paper compares Drug-GPT 3, Drug-GPT 4, and ChatGPT in healthcare Q&A, showing that specialized Drug-GPT models provide more targeted and in-depth insights than the general-purpose ChatGPT.

Contribution

It introduces a comparative framework for evaluating healthcare-specific GPT models versus general models in terms of accuracy and relevance.

Findings

Drug-GPT models offer more targeted insights.

ChatGPT provides broader, high-level responses.

Specialized models outperform general models in depth.

Abstract

This study presents a comparative analysis of three Generative Pre-trained Transformer (GPT) solutions in a question and answer (Q&A) setting: Drug-GPT 3, Drug-GPT 4, and ChatGPT, in the context of healthcare applications. The objective is to determine which model delivers the most accurate and relevant information in response to prompts related to patient experiences with atopic dermatitis (AD) and healthcare professional (HCP) discussions about diabetes. The results demonstrate that while all three models are capable of generating relevant and accurate responses, Drug-GPT 3 and Drug-GPT 4, which are supported by curated datasets of patient and HCP social media and message board posts, provide more targeted and in-depth insights. ChatGPT, a more general-purpose model, generates broader and more general responses, which may be valuable for readers seeking a high-level understanding of the topics but may lack the depth and personal insights found in the answers generated by the specialized Drug-GPT models. This comparative analysis highlights the importance of considering the language model's perspective, depth of knowledge, and currency when evaluating the usefulness of generated information in healthcare applications.