WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models

TL;DR

This paper compares fine-tuning and in-context learning approaches using large language models for automatic clinical note generation from doctor-patient conversations, achieving top rankings and promising human preference results.

Contribution

It introduces and evaluates two approaches—fine-tuning PLMs and using ICL with LLMs—for clinical note generation, demonstrating superior performance and human preference for ICL-generated notes.

Findings

ICL with GPT-4 outperforms fine-tuned models in quality.

Both approaches achieve high automatic metric scores.

ICL-generated notes are often preferred by experts.

Abstract

This paper describes our submission to the MEDIQA-Chat 2023 shared task for automatic clinical note generation from doctor-patient conversations. We report results for two approaches: the first fine-tunes a pre-trained language model (PLM) on the shared task data, and the second uses few-shot in-context learning (ICL) with a large language model (LLM). Both achieve high performance as measured by automatic metrics (e.g. ROUGE, BERTScore) and ranked second and first, respectively, of all submissions to the shared task. Expert human scrutiny indicates that notes generated via the ICL-based approach with GPT-4 are preferred about as often as human-written notes, making it a promising path toward automated note generation from doctor-patient conversations.