Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension

TL;DR

This paper evaluates GPT's performance on biomedical machine reading comprehension benchmarks, introduces a novel prompting method called Implicit RAG, and demonstrates GPT's competitive results surpassing supervised models in some cases.

Contribution

The study introduces a new prompting strategy, Implicit RAG, to improve GPT's biomedical MRC performance and provides comprehensive evaluation across multiple datasets.

Findings

GPT achieves state-of-the-art results on two datasets.

The novel prompting method outperforms conventional prompts in two datasets.

GPT can outperform supervised models even in zero-shot settings.

Abstract

Large language models (LLMs) have shown remarkable performance on many tasks in different domains. However, their performance in closed-book biomedical machine reading comprehension (MRC) has not been evaluated in depth. In this work, we evaluate GPT on four closed-book biomedical MRC benchmarks. We experiment with different conventional prompting techniques as well as introduce our own novel prompting method. To solve some of the retrieval problems inherent to LLMs, we propose a prompting strategy named Implicit Retrieval Augmented Generation (RAG) that alleviates the need for using vector databases to retrieve important chunks in traditional RAG setups. Moreover, we report qualitative assessments on the natural language generation outputs from our approach. The results show that our new prompting technique is able to get the best performance in two out of four datasets and ranks second in rest of them. Experiments show that modern-day LLMs like GPT even in a zero-shot setting can outperform supervised models, leading to new state-of-the-art (SoTA) results on two of the benchmarks.