UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?

TL;DR

This paper introduces a novel LLM cooperation system called a doctor-patient loop for generating high-quality synthetic medical conversations, demonstrating promising results and comparative advantages over ChatGPT and GPT-4.

Contribution

The paper proposes a new LLM cooperation framework for synthetic medical dialogue generation, enhancing data quality for medical NLP tasks.

Findings

The system achieves reasonable performance on automatic metrics.

Comparative analysis shows advantages over ChatGPT and GPT-4.

The approach effectively utilizes cooperation among LLMs for data generation.

Abstract

This paper presents UMASS_BioNLP team participation in the MEDIQA-Chat 2023 shared task for Task-A and Task-C. We focus especially on Task-C and propose a novel LLMs cooperation system named a doctor-patient loop to generate high-quality conversation data sets. The experiment results demonstrate that our approaches yield reasonable performance as evaluated by automatic metrics such as ROUGE, medical concept recall, BLEU, and Self-BLEU. Furthermore, we conducted a comparative analysis between our proposed method and ChatGPT and GPT-4. This analysis also investigates the potential of utilizing cooperation LLMs to generate high-quality datasets.