# ClinVLA: an image-text retrieval method for promoting hospital diagnosis data analysis and patient health prediction

## TL;DR

ClinVLA is a new method that improves the alignment of medical images and text, enhancing hospital diagnosis and patient health prediction.

## Contribution

ClinVLA introduces a novel multi-view input design and adapter module for better medical image-text alignment and task transfer efficiency.

## Key findings

- ClinVLA improves text-to-image retrieval accuracy by over 3% compared to similar algorithms.
- The model increases image-to-text retrieval accuracy by approximately 5%.
- It ensures semantic consistency between images and texts using global and local alignment losses.

## Abstract

Medical visual-language alignment plays an important role in hospital diagnostic data analysis and patient health prediction. However, existing multimodal alignment models, such as CLIP, while performing well in some tasks, often fail to accurately capture the fine-grained alignment between complex medical images and texts, and lack the capability to handle multi-view radiological image inputs. To address these issues, this paper proposes the ClinVLA model, an efficient visual-language alignment method. Specifically, ClinVLA enhances image feature representation through an innovative multi-view input design, including both frontal and lateral views. Furthermore, ClinVLA introduces an innovative adapter module, making the model more efficient in task transfer and language transformation, significantly improving performance in cross-modal learning. Finally, by incorporating both global and local alignment losses, ClinVLA ensures semantic consistency between images and texts, optimizing the accuracy and efficiency of image-text matching. Experimental results on datasets such as CheXpert and RSNA Pneumonia show that ClinVLA improves text-to-image retrieval accuracy by over 3% compared to the best-performing similar algorithms, and increases image-to-text retrieval accuracy by approximately 5%. ClinVLA provides a new solution for medical image analysis, with broad application prospects.