OrthoInsight: Rib Fracture Diagnosis and Report Generation Based on Multi-Modal Large Models

TL;DR

OrthoInsight is a multi-modal deep learning system that automates rib fracture detection and report generation from CT scans, integrating visual detection, clinical knowledge, and language models to assist radiologists.

Contribution

It introduces a novel multi-modal framework combining detection, knowledge retrieval, and report generation for rib fractures, outperforming existing models.

Findings

Achieved high diagnostic accuracy and report quality on large CT dataset.

Outperformed GPT-4 and Claude-3 in clinical report generation.

Demonstrated effective integration of visual and textual data in medical diagnosis.

Abstract

The growing volume of medical imaging data has increased the need for automated diagnostic tools, especially for musculoskeletal injuries like rib fractures, commonly detected via CT scans. Manual interpretation is time-consuming and error-prone. We propose OrthoInsight, a multi-modal deep learning framework for rib fracture diagnosis and report generation. It integrates a YOLOv9 model for fracture detection, a medical knowledge graph for retrieving clinical context, and a fine-tuned LLaVA language model for generating diagnostic reports. OrthoInsight combines visual features from CT images with expert textual data to deliver clinically useful outputs. Evaluated on 28,675 annotated CT images and expert reports, it achieves high performance across Diagnostic Accuracy, Content Completeness, Logical Coherence, and Clinical Guidance Value, with an average score of 4.28, outperforming models like GPT-4 and Claude-3. This study demonstrates the potential of multi-modal learning in transforming medical image analysis and providing effective support for radiologists.