XAI4LLM. Let Machine Learning Models and LLMs Collaborate for Enhanced In-Context Learning in Healthcare

TL;DR

This paper introduces a knowledge-guided in-context learning framework for large language models to improve clinical decision support by processing structured data, enhancing fairness, and enabling zero-shot deployment, with a comprehensive evaluation across multiple prompt designs.

Contribution

The study presents the first systematic analysis of ICL design for clinical tabular tasks using LLMs, integrating domain knowledge and comparing performance with classical ML models.

Findings

LLMs with narrative prompts improve recall and reduce gender bias.

Traditional ML models outperform LLMs in balanced precision-recall scenarios.

LLMs offer advantages like zero-shot deployment and increased fairness despite higher latency.

Abstract

Clinical decision support systems require models that are not only highly accurate but also equitable and sensitive to the implications of missed diagnoses. In this study, we introduce a knowledge-guided in-context learning (ICL) framework designed to enable large language models (LLMs) to effectively process structured clinical data. Our approach integrates domain-specific feature groupings, carefully balanced few-shot examples, and task-specific prompting strategies. We systematically evaluate this method across seventy distinct ICL designs by various prompt variations and two different communication styles-natural-language narrative and numeric conversational-and compare its performance to robust classical machine learning (ML) benchmarks on tasks involving heart disease and diabetes prediction. Our findings indicate that while traditional ML models maintain superior performance in balanced precision-recall scenarios, LLMs employing narrative prompts with integrated domain knowledge achieve higher recall and significantly reduce gender bias, effectively narrowing fairness disparities by an order of magnitude. Despite the current limitation of increased inference latency, LLMs provide notable advantages, including the capacity for zero-shot deployment and enhanced equity. This research offers the first comprehensive analysis of ICL design considerations for applying LLMs to tabular clinical tasks and highlights distillation and multimodal extensions as promising directions for future research.