Leveraging Open-Source Large Language Models for encoding Social Determinants of Health using an Intelligent Router

TL;DR

This paper presents an intelligent routing system that directs clinical notes to open-source large language models optimized for specific Social Determinants of Health coding tasks, achieving state-of-the-art accuracy and addressing privacy concerns.

Contribution

The authors introduce a novel intelligent routing architecture that leverages multiple open-source LLMs for SDOH coding, outperforming closed models like GPT-4o.

Findings

Achieved 96.4% accuracy across 13 SDOH codes.

Outperformed closed-source models such as GPT-4o.

Utilized synthetic data generation to enhance training without privacy risks.

Abstract

Social Determinants of Health (SDOH), also known as Health-Related Social Needs (HSRN), play a significant role in patient health outcomes. The Centers for Disease Control and Prevention (CDC) introduced a subset of ICD-10 codes called Z-codes to recognize and measure SDOH. However, Z-codes are infrequently coded in a patient's Electronic Health Record (EHR), and instead, in many cases, need to be inferred from clinical notes. Previous research has shown that large language models (LLMs) show promise on extracting unstructured data from EHRs, but it can be difficult to identify a single model that performs best on varied coding tasks. Further, clinical notes contain protected health information posing a challenge for the use of closed-source language models from commercial vendors. The identification of open-source LLMs that can be run within health organizations and exhibit high performance on SDOH tasks is an important issue to solve. Here, we introduce an intelligent routing system for SDOH coding that uses a language model router to direct medical record data to open-source LLMs that demonstrate optimal performance on specific SDOH codes. This intelligent routing system exhibits state of the art performance of 96.4% accuracy averaged across 13 codes, including homelessness and food insecurity, outperforming closed models such as GPT-4o. We leveraged a publicly-available, deidentified dataset of medical record notes to run the router, but we also introduce a synthetic data generation and validation paradigm to increase the scale of training data without needing privacy-protected medical records. Together, we demonstrate an architecture for intelligent routing of inputs to task-optimal language models to achieve high performance across a set of medical coding sub-tasks.