Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints

TL;DR

This paper evaluates GPT-3.5 and GPT-4's ability to identify and correct medical errors in clinical notes, proposing prompting strategies that incorporate error-span hints to improve accuracy, with implications for clinical deployment.

Contribution

It introduces novel prompting strategies that integrate error-span predictions from smaller models to enhance LLM correction capabilities in medical texts.

Findings

Prompting strategies significantly improve correction accuracy.

Best solution ranked sixth in the shared task leaderboard.

Analysis reveals factors affecting LLM correction performance.

Abstract

The MEDIQA-CORR 2024 shared task aims to assess the ability of Large Language Models (LLMs) to identify and correct medical errors in clinical notes. In this study, we evaluate the capability of general LLMs, specifically GPT-3.5 and GPT-4, to identify and correct medical errors with multiple prompting strategies. Recognising the limitation of LLMs in generating accurate corrections only via prompting strategies, we propose incorporating error-span predictions from a smaller, fine-tuned model in two ways: 1) by presenting it as a hint in the prompt and 2) by framing it as multiple-choice questions from which the LLM can choose the best correction. We found that our proposed prompting strategies significantly improve the LLM's ability to generate corrections. Our best-performing solution with 8-shot + CoT + hints ranked sixth in the shared task leaderboard. Additionally, our comprehensive analyses show the impact of the location of the error sentence, the prompted role, and the position of the multiple-choice option on the accuracy of the LLM. This prompts further questions about the readiness of LLM to be implemented in real-world clinical settings.