Can Public LLMs be used for Self-Diagnosis of Medical Conditions ?

TL;DR

This paper evaluates the potential of public large language models like GPT-4.0 and Gemini for self-diagnosis of medical conditions, highlighting their performance differences, limitations, and the impact of retrieval-augmented generation.

Contribution

It provides a comparative analysis of GPT-4.0 and Gemini on a large self-diagnosis dataset and discusses their limitations and potential in healthcare applications.

Findings

GPT-4.0 achieved 63.07% accuracy in self-diagnosis.

Gemini achieved only 6.01% accuracy.

Retrieval-augmented generation improves diagnostic performance.

Abstract

Advancements in deep learning have generated a large-scale interest in the development of foundational deep learning models. The development of Large Language Models (LLM) has evolved as a transformative paradigm in conversational tasks, which has led to its integration and extension even in the critical domain of healthcare. With LLMs becoming widely popular and their public access through open-source models and integration with other applications, there is a need to investigate their potential and limitations. One such crucial task where LLMs are applied but require a deeper understanding is that of self-diagnosis of medical conditions based on bias-validating symptoms in the interest of public health. The widespread integration of Gemini with Google search and GPT-4.0 with Bing search has led to a shift in the trend of self-diagnosis using search engines to conversational LLM models. Owing to the critical nature of the task, it is prudent to investigate and understand the potential and limitations of public LLMs in the task of self-diagnosis. In this study, we prepare a prompt engineered dataset of 10000 samples and test the performance on the general task of self-diagnosis. We compared the performance of both the state-of-the-art GPT-4.0 and the fee Gemini model on the task of self-diagnosis and recorded contrasting accuracies of 63.07% and 6.01%, respectively. We also discuss the challenges, limitations, and potential of both Gemini and GPT-4.0 for the task of self-diagnosis to facilitate future research and towards the broader impact of general public knowledge. Furthermore, we demonstrate the potential and improvement in performance for the task of self-diagnosis using Retrieval Augmented Generation.