SocialLM: Social Signal Processing of Patient-Provider Communication using LLMs and Contextual Aggregation

TL;DR

This paper explores using large language models to detect social behaviors in clinical transcripts without fine-tuning, addressing variability issues with an ensemble method to improve accuracy and stability.

Contribution

It introduces an agreement-weighted ensemble approach that enhances social signal detection in clinical conversations using LLMs under API constraints.

Findings

LLMs can reliably detect social signals in clinical transcripts.

Performance varies by patient race and visit segment.

Ensemble method improves accuracy and stability.

Abstract

Effective patient-provider communication is difficult to assess at scale. We examine whether large language models (LLMs) can track 20 social behaviors from clinical transcripts without fine-tuning. Across three model families and multiple prompting strategies, LLMs reliably detect social signals, though performance varies by patient race and visit segment. To address this variability under query-only API constraints, we introduce an agreement-weighted ensemble using group-level agreement patterns. This approach improves both accuracy and stability over the best individual model, demonstrating a practical pathway for scalable social signal tracking in clinical conversations.