Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models

TL;DR

This paper introduces ClinRaGen, a method that enhances small language models with large language model reasoning and domain knowledge to generate trustworthy, multimodal rationales for disease diagnosis, achieving state-of-the-art results.

Contribution

ClinRaGen presents a novel framework combining rationale distillation and knowledge injection to improve multimodal reasoning and interpretability in small language models for clinical diagnosis.

Findings

ClinRaGen outperforms existing models in disease diagnosis accuracy.

It produces more human-understandable and trustworthy rationales.

The approach effectively integrates multimodal data with domain knowledge.

Abstract

Interpretation is critical for disease diagnosis, but existing models struggle to balance predictive accuracy with human-understandable rationales. While large language models (LLMs) offer strong reasoning abilities, their clinical use is limited by high computational costs and restricted multimodal reasoning ability. Small language models (SLMs) are efficient but lack advanced reasoning for integrating multimodal medical data. In addition, both LLMs and SLMs lack domain knowledge for trustworthy reasoning. Therefore, we propose ClinRaGen, enhancing SLMs by leveraging LLM-derived reasoning ability via rationale distillation and domain knowledge injection for trustworthy multimodal rationale generation. Key innovations include a sequential rationale distillation framework that equips SLMs with LLM-comparable multimodal reasoning abilities, and a knowledge-augmented attention mechanism that jointly unifies multimodal representation from time series and textual data in the same encoding space, enabling it to be naturally interpreted by SLMs while incorporating domain knowledge for reliable rationale generation. Experiments on real-world medical datasets show that ClinRaGen achieves state-of-the-art performance in disease diagnosis and rationale generation, demonstrating the effectiveness of combining LLM-driven reasoning with knowledge augmentation for improved interpretability.