Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing

TL;DR

This paper introduces a method that fine-tunes large language models to generate weakly-labeled data for clinical NLP tasks, significantly reducing the need for extensive annotated datasets while maintaining high performance.

Contribution

It presents a novel approach combining fine-tuned LLMs with weak supervision to improve clinical NLP without relying on large domain-specific labeled data.

Findings

Weakly supervised models outperform traditional models with minimal gold data.

Using only 10 gold notes, models outperform PubMedBERT by up to 47.9% in F1 score.

Close to full performance achieved with just 50 gold notes.

Abstract

The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7% to 47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.