Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting

TL;DR

This paper introduces a privacy-preserving method that uses large language models to generate contextual prompts, significantly improving small medical language models' performance in sensitive healthcare tasks.

Contribution

The paper proposes a novel approach leveraging LLMs to generate medical context prompts, enhancing small model performance while maintaining data privacy in healthcare applications.

Findings

Up to 22.57% accuracy improvement in medical tasks

Achieved state-of-the-art results in privacy-restricted scenarios

Demonstrated generalizability across domains

Abstract

Large language models (LLMs) demonstrate remarkable medical expertise, but data privacy concerns impede their direct use in healthcare environments. Although offering improved data privacy protection, domain-specific small language models (SLMs) often underperform LLMs, emphasizing the need for methods that reduce this performance gap while alleviating privacy concerns. In this paper, we present a simple yet effective method that harnesses LLMs' medical proficiency to boost SLM performance in medical tasks under privacy-restricted scenarios. Specifically, we mitigate patient privacy issues by extracting keywords from medical data and prompting the LLM to generate a medical knowledge-intensive context by simulating clinicians' thought processes. This context serves as additional input for SLMs, augmenting their decision-making capabilities. Our method significantly enhances performance in both few-shot and full training settings across three medical knowledge-intensive tasks, achieving up to a 22.57% increase in absolute accuracy compared to SLM fine-tuning without context, and sets new state-of-the-art results in two medical tasks within privacy-restricted scenarios. Further out-of-domain testing and experiments in two general domain datasets showcase its generalizability and broad applicability. Our code can be found at https://github.com/XZhang97666/PrivacyBoost-SLM.