Enhancing Medical Cross-Modal Hashing Retrieval using Dropout-Voting Mixture-of-Experts Fusion

TL;DR

This paper introduces MCMFH, a novel medical cross-modal hashing retrieval framework that combines dropout voting, mixture-of-experts fusion, and hybrid loss to improve accuracy and speed in low-memory medical data environments.

Contribution

The paper presents a new framework integrating dropout voting, MoE fusion, and hybrid loss for enhanced medical cross-modal hashing retrieval performance.

Findings

Achieves high accuracy in medical image-text retrieval

Enables fast retrieval in low-memory settings

Demonstrates effectiveness on radiological and non-radiological datasets

Abstract

In recent years, cross-modal retrieval using images and text has become an active area of research, especially in the medical domain. The abundance of data in various modalities in this field has led to a growing importance of cross-modal retrieval for efficient image interpretation, data-driven diagnostic support, and medical education. In the context of the increasing integration of distributed medical data across healthcare facilities with the objective of enhancing interoperability, it is imperative to optimize the performance of retrieval systems in terms of the speed, memory efficiency, and accuracy of the retrieved data. This necessity arises in response to the substantial surge in data volume that characterizes contemporary medical practices. In this study, we propose a novel framework that incorporates dropout voting and mixture-of-experts (MoE) based contrastive fusion modules into a CLIP-based cross-modal hashing retrieval structure. We also propose the application of hybrid loss. So we now call our model MCMFH which is a medical cross-modal fusion hashing retrieval. Our method enables the simultaneous achievement of high accuracy and fast retrieval speed in low-memory environments. The model is demonstrated through experiments on radiological and non-radiological medical datasets.