TotalFM: An Organ-Separated Framework for 3D-CT Vision Foundation Models

TL;DR

TotalFM introduces an organ-separated framework for 3D-CT vision models that improves efficiency and generalization in clinical tasks by leveraging large-scale data, segmentation, and contrastive learning.

Contribution

The paper presents a novel organ-separated learning framework for 3D-CT foundation models, combining segmentation, large language models, and contrastive learning for improved clinical task performance.

Findings

Outperforms CT-CLIP and Merlin in zero-shot organ-wise lesion classification.

Achieves higher AUROC in zero-shot finding-wise lesion classification.

Comparable performance to existing Vision-Language Models in report generation.

Abstract

While foundation models in radiology are expected to be applied to various clinical tasks, computational cost constraints remain a major challenge when training on 3D-CT volumetric data. In this study, we propose TotalFM, a radiological foundation model that efficiently learns the correspondence between 3D-CT images and linguistic expressions based on the concept of organ separation, utilizing a large-scale dataset of 140,000 series. By automating the creation of organ volume and finding-sentence pairs through segmentation techniques and Large Language Model (LLM)-based radiology report processing, and by combining self-supervised pre-training via VideoMAE with contrastive learning using volume-text pairs, we aimed to balance computational efficiency and representation capability. In zero-shot organ-wise lesion classification tasks, the proposed model achieved higher F1 scores in 83% (5/6) of organs compared to CT-CLIP and 64% (9/14) of organs compared to Merlin. These results suggest that the proposed model exhibits high generalization performance in a clinical evaluation setting using actual radiology report sentences. Furthermore, in zero-shot finding-wise lesion classification tasks, our model achieved a higher AUROC in 83% (25/30) of finding categories compared to Merlin. We also confirmed performance comparable to existing Vision-Language Models (VLMs) in radiology report generation tasks. Our results demonstrate that the organ-separated learning framework can serve as a realistic and effective design guideline for the practical implementation of 3D-CT foundation models.