Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Model

TL;DR

This study demonstrates that GPT-3.5 can effectively extract demographics, social determinants, and family history from clinical notes using zero-shot learning with minimal instructions, evaluated with both traditional and semantic metrics.

Contribution

The paper introduces a zero-shot learning approach using GPT-3.5 for extracting health-related information from clinical notes with minimal guidance, a novel application in medical NLP.

Findings

GPT-3.5 achieved high F1 scores in demographics extraction (0.975).

Semantic similarity metrics provide additional insights into extraction performance.

Limitations of GPT models were identified, guiding future improvements.

Abstract

Demographics, Social determinants of health, and family history documented in the unstructured text within the electronic health records are increasingly being studied to understand how this information can be utilized with the structured data to improve healthcare outcomes. After the GPT models were released, many studies have applied GPT models to extract this information from the narrative clinical notes. Different from the existing work, our research focuses on investigating the zero-shot learning on extracting this information together by providing minimum information to the GPT model. We utilize de-identified real-world clinical notes annotated for demographics, various social determinants, and family history information. Given that the GPT model might provide text different from the text in the original data, we explore two sets of evaluation metrics, including the traditional NER evaluation metrics and semantic similarity evaluation metrics, to completely understand the performance. Our results show that the GPT-3.5 method achieved an average of 0.975 F1 on demographics extraction, 0.615 F1 on social determinants extraction, and 0.722 F1 on family history extraction. We believe these results can be further improved through model fine-tuning or few-shots learning. Through the case studies, we also identified the limitations of the GPT models, which need to be addressed in future research.