Vision-Language Model Based Multi-Expert Fusion for CT Image Classification

TL;DR

This paper introduces a multi-expert, source-aware framework combining lung-aware 3D, MedSigLIP, and Transformer experts for robust COVID-19 CT classification across multiple sources, achieving high accuracy and AUC.

Contribution

It presents a novel three-stage multi-expert fusion approach that explicitly models source information to improve multi-source COVID-19 CT classification robustness.

Findings

Stage 1 model achieved macro-F1 of 0.9711

Stage 2 experts achieved AUC scores over 0.985

Source classifier reached over 91% accuracy

Abstract

Robust detection of COVID-19 from chest CT remains challenging in multi-institutional settings due to substantial source shift, source imbalance, and hidden test-source identities. In this work, we propose a three-stage source-aware multi-expert framework for multi-source COVID-19 CT classification. First, we build a lung-aware 3D expert by combining original CT volumes and lung-extracted CT volumes for volumetric classification. Second, we develop two MedSigLIP-based experts: a slice-wise representation and probability learning module, and a Transformer-based inter-slice context modeling module for capturing cross-slice dependency. Third, we train a source classifier to predict the latent source identity of each test scan. By leveraging the predicted source information, we perform model fusion and voting based on different experts. On the validation set covering all four sources, the Stage 1 model achieves the best macro-F1 of 0.9711, ACC of 0.9712, and AUC of 0.9791. Stage~2a and Stage~2b achieve the best AUC scores of 0.9864 and 0.9854, respectively. Stage~3 source classifier reaches 0.9107 ACC and 0.9114 F1. These results demonstrate that source-aware expert modeling and hierarchical voting provide an effective solution for robust COVID-19 CT classification under heterogeneous multi-source conditions.