Task formulation for Extracting Social Determinants of Health from Clinical Narratives

TL;DR

This paper compares three different NLP systems developed for extracting social determinants of health from clinical narratives, highlighting their methodologies, performance, and practical trade-offs in a challenging shared task.

Contribution

It introduces and evaluates three distinct approaches—machine learning classifiers, large language models, and rule-based extraction—for SDOH information retrieval in clinical texts.

Findings

The systems achieved F1 scores of 0.884, 0.831, and 0.663.

Independent phrase extraction outperformed relation-based methods.

Large language models provided a good balance of performance and flexibility.

Abstract

Objective: The 2022 n2c2 NLP Challenge posed identification of social determinants of health (SDOH) in clinical narratives. We present three systems that we developed for the Challenge and discuss the distinctive task formulation used in each of the three systems. Materials and Methods: The first system identifies target pieces of information independently using machine learning classifiers. The second system uses a large language model (LLM) to extract complete structured outputs per document. The third system extracts candidate phrases using machine learning and identifies target relations with hand-crafted rules. Results: The three systems achieved F1 scores of 0.884, 0.831, and 0.663 in the Subtask A of the Challenge, which are ranked third, seventh, and eighth among the 15 participating teams. The review of the extraction results from our systems reveals characteristics of each approach and those of the SODH extraction task. Discussion: Phrases and relations annotated in the task is unique and diverse, not conforming to the conventional event extraction task. These annotations are difficult to model with limited training data. The system that extracts information independently, ignoring the annotated relations, achieves the highest F1 score. Meanwhile, LLM with its versatile capability achieves the high F1 score, while respecting the annotated relations. The rule-based system tackling relation extraction obtains the low F1 score, while it is the most explainable approach. Conclusion: The F1 scores of the three systems vary in this challenge setting, but each approach has advantages and disadvantages in a practical application. The selection of the approach depends not only on the F1 score but also on the requirements in the application.