VOILA: Complexity-Aware Universal Segmentation of CT images by Voxel Interacting with Language

TL;DR

VOILA introduces a novel complexity-aware, language-interacting framework for universal CT image segmentation, effectively handling class imbalance and improving generalizability across datasets with reduced computational costs.

Contribution

The paper presents a new voxel-language interaction approach combined with complexity-aware sampling to enhance universal CT segmentation performance and generalizability.

Findings

Improved segmentation accuracy across multiple datasets.

Reduced training parameters and computational cost.

Enhanced handling of class imbalance and target volume variations.

Abstract

Satisfactory progress has been achieved recently in universal segmentation of CT images. Following the success of vision-language methods, there is a growing trend towards utilizing text prompts and contrastive learning to develop universal segmentation models. However, there exists a significant imbalance in information density between 3D images and text prompts. Moreover, the standard fully connected layer segmentation approach faces significant challenges in handling multiple classes and exhibits poor generalizability. To address these challenges, we propose the VOxel Interacting with LAnguage method (VOILA) for universal CT image segmentation. Initially, we align voxels and language into a shared representation space and classify voxels on the basis of cosine similarity. Subsequently, we develop the Voxel-Language Interaction framework to mitigate the impact of class imbalance caused by foreground-background discrepancies and variations in target volumes. Furthermore, a Complexity-Aware Sampling method is proposed to focus on region hard to segment, achieved by generating pseudo-heatmaps from a trainable Gaussian mixture distribution. Our results indicate the proposed VOILA is capable to achieve improved performance with reduced parameters and computational cost during training. Furthermore, it demonstrates significant generalizability across diverse datasets without additional fine-tuning.