Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding

TL;DR

This study evaluates the performance of large language models like GPT-3.5, GPT-4, and Bard on various clinical language understanding tasks, introducing a novel prompting method to improve healthcare-related NLP performance.

Contribution

The paper presents a comprehensive evaluation of state-of-the-art LLMs in clinical NLP and introduces self-questioning prompting (SQP) to enhance their effectiveness in healthcare tasks.

Findings

SQP improves LLM performance on clinical tasks

Error analysis reveals key challenges in relation extraction

Task-specific prompting strategies are crucial for healthcare NLP

Abstract

Large language models (LLMs) have made significant progress in various domains, including healthcare. However, the specialized nature of clinical language understanding tasks presents unique challenges and limitations that warrant further investigation. In this study, we conduct a comprehensive evaluation of state-of-the-art LLMs, namely GPT-3.5, GPT-4, and Bard, within the realm of clinical language understanding tasks. These tasks span a diverse range, including named entity recognition, relation extraction, natural language inference, semantic textual similarity, document classification, and question-answering. We also introduce a novel prompting strategy, self-questioning prompting (SQP), tailored to enhance LLMs' performance by eliciting informative questions and answers pertinent to the clinical scenarios at hand. Our evaluation underscores the significance of task-specific learning strategies and prompting techniques for improving LLMs' effectiveness in healthcare-related tasks. Additionally, our in-depth error analysis on the challenging relation extraction task offers valuable insights into error distribution and potential avenues for improvement using SQP. Our study sheds light on the practical implications of employing LLMs in the specialized domain of healthcare, serving as a foundation for future research and the development of potential applications in healthcare settings.