Double-mix pseudo-label framework: enhancing semi-supervised segmentation on category-imbalanced CT volumes

TL;DR

This paper introduces a new method to improve CT image segmentation with limited labeled data by focusing on challenging categories.

Contribution

The novel Double-Mix Pseudo-label Framework and confidence-difficulty weight allocation method improve semi-supervised segmentation.

Findings

The method achieved a 5.1% improvement in Dice score for CHD dataset with 5% labeled data.

It achieved a 7.0% improvement in Dice score for BTCV dataset with 40% labeled data.

The approach enhances segmentation of difficult categories in CT volumes.

Abstract

Deep-learning-based supervised CT segmentation relies on fully and densely labeled data, the labeling process of which is time-consuming. In this study, our proposed method aims to improve segmentation performance on CT volumes with limited annotated data by considering category-wise difficulties and distribution.

We propose a novel confidence-difficulty weight (CDifW) allocation method that considers confidence levels, balancing the training across different categories, influencing the loss function and volume-mixing process for pseudo-label generation. Additionally, we introduce a novel Double-Mix Pseudo-label Framework (DMPF), which strategically selects categories for image blending based on the distribution of voxel-counts per category and the weight of segmentation difficulty. DMPF is designed to enhance the segmentation performance of categories that are challenging to segment.

Our approach was tested on two commonly used datasets: a Congenital Heart Disease (CHD) dataset and a Beyond-the-Cranial-Vault (BTCV) Abdomen dataset. Compared to the SOTA methods, our approach achieved an improvement of 5.1% and 7.0% in Dice score for the segmentation of difficult-to-segment categories on 5% of the labeled data in CHD and 40% of the labeled data in BTCV, respectively.

Our method improves segmentation performance in difficult categories within CT volumes by category-wise weights and weight-based mixture augmentation. Our method was validated across multiple datasets and is significant for advancing semi-supervised segmentation tasks in health care. The code is available at https://github.com/MoriLabNU/Double-Mix.

Figures

The category-wise voxel-count and Dice score of the fully-supervised segmentation on BTCV [[2](#CR2), [9](#CR9)] using V-Net [[10](#CR10)]. a Voxel-counts, b Dice scores

The category-wise voxel-count and Dice score of the fully-supervised segmentation on BTCV [[2](#CR2), [9](#CR9)] using V-Net [[10](#CR10)]. a Voxel-counts, b Dice scores

The category-wise difficulty evaluation during model training. a Based on Dice score [2], b based on confidence. The weights based on confidence are smoother than those based on Dice, which is particularly noticeable in small categories, such as PA (vibrant orange, dashed) and LAG (muted teal, dashed)

Overview of the proposed DMPFFig. 4Overview of the proposed single DMP module. a Category mask $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{M}$$\end{document}$ generation. b Data Mixture

Comparative experiments with other methods using 40% BTCV dataset. Some high-difficulty categories are highlighted by red frames Fig. 6Comparative experiments with other methods using 5% of the CHD dataset

Comparative experiments with other methods using 40% BTCV dataset. Some high-difficulty categories are highlighted by red frames Fig. 6Comparative experiments with other methods using 5% of the CHD dataset

Results using different percentages of labeled data. a BTCV, b CHDFig. 8The result of Wilcoxon signed-rank test. a The results using 10%, 20%, and 40% of the BTCV dataset as the labeled data, b the results using 5%, 10%, and 20% of the CHD dataset as the labeled data

Results using different percentages of labeled data. a BTCV, b CHDFig. 8The result of Wilcoxon signed-rank test. a The results using 10%, 20%, and 40% of the BTCV dataset as the labeled data, b the results using 5%, 10%, and 20% of the CHD dataset as the labeled data

Dice and ASD results from three-fold cross-validation on colon dataset. a Dice scores, b ASDs Fig. 10Segmentation results on balanced colon segmentation dataset. The arrow indicates a potential erroneous segmentation likely due to the disruption of spatial information by ClassMix

Dice and ASD results from three-fold cross-validation on colon dataset. a Dice scores, b ASDs Fig. 10Segmentation results on balanced colon segmentation dataset. The arrow indicates a potential erroneous segmentation likely due to the disruption of spatial information by ClassMix

Full text

Introduction

Segmentation of organs and tissues from CT volumes is a crucial component of the computer-aided diagnosis paradigm, while the annotation process requires expert knowledge and is time-consuming. To address this issue, Semi-Supervised Learning (SSL), which utilizes both limited annotated and extensive unannotated data, has shown promising results in CT segmentation [1–3].

In SSL, given the limited amount of labeled data, employing sample mixtures is widely used. CutMix [4] and ClassMix [5] enhance images by cutting and merging labeled and unlabeled images. CutMix utilizes annotated data, whereas ClassMix uses pseudo-labels to identify areas for merging. The category-wise voxel-count distribution in medical image segmentation is often highly imbalanced, as shown in Fig. 1a. The liver’s voxel-count is over 500 times higher than left-adrenal-gland, leading to a large difference in Dice score in full-supervision segmentation results in Fig. 1b. We defined these voxel-count differences as distribution differences. These mixture methods do not consider category-wise distribution imbalance, resulting in some categories with few voxels being easily overlooked during merging and diminishing segmentation performance for these categories.Fig. 1. The category-wise voxel-count and Dice score of the fully-supervised segmentation on BTCV [2, 9] using V-Net [10]. a Voxel-counts, b Dice scores

Another method for SSL is using the co-model-training framework such as cross-pseudo-supervision (CPS) [6], concurrently training two models to achieve similar outputs. It still faces category imbalance during training. Lin et al. [3] addressed category bias by applying category-wise distribution-based weight during co-model training based on CPS. However, they did not consider the heterogeneity of the two models in co-model training. Related work [7] shows that to achieve enhanced generalization capabilities, individual models should be both accurate and heterogeneous. As described in related work [2], it is necessary to increase the heterogeneity between the two models during co-model training. They compute difficulty weights based on Dice scores and voxel-count distribution weights for co-model training within the CPS framework, improving model performance by increasing the heterogeneity between the two models. However, when employing category imbalance datasets, the Dice score-based difficulty weight may have notable fluctuations during the training process, as illustrated in Fig. 2a. For some low voxel-count categories, even small pixel-level prediction errors can significantly impact the performance metrics for these small categories, leading to huge variations in the Dice score in the training process. This variability causes fluctuations in training, which affect the performance of the model. To address this issue, the confidence in the model’s output for each category, exhibiting the category-wise uncertainty, may provide a more stable metric for assessing the difficulty of each category [8]. As shown in Fig. 2b, confidence changes smoother during training, and the differences in evaluated difficulty among various categories become more pronounced. However, using only confidence as a measure for difficulty weights may not be sensitive to the categories with high changing speed in difficulty during the training process. Therefore, when estimating the category-wise difficulty, both category-wise confidence and Dice values should be taken into consideration.Fig. 2. The category-wise difficulty evaluation during model training. a Based on Dice score [2], b based on confidence. The weights based on confidence are smoother than those based on Dice, which is particularly noticeable in small categories, such as PA (vibrant orange, dashed) and LAG (muted teal, dashed)

In this work, we introduce a comprehensive approach to address category imbalance in semi-supervised CT volume segmentation. The primary contributions are reflected in the following aspects: (i) Introduction of confidence-difficulty weight (CDifW): We propose the CDifW, a novel category-wise difficulty weight that integrates both confidence and Dice score. We also incorporate the distribution weight (DisW), calculated from category-wise voxel-counts, to further refine our method. (ii) Novel Double-Mix Pseudo-label (DMP) Module: To tackle the limitations of existing data augmentation strategies, we introduce the DMP module, which uses these weights to focus augmentation efforts on high-difficulty categories. (iii) Innovative Double-Mix Pseudo-label Framework (DMPF): We propose the DMPF. This novel co-model training framework employs a CPS approach to enhance focus on distinct category-wise differences.

By applying the CDifW and DisW within both the DMP module and the overall model training, our method aims to improve the segmentation performance of imbalanced categories in CT volumes.Fig. 3. Overview of the proposed DMPFFig. 4Overview of the proposed single DMP module. a Category mask \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{M}$$\end{document} generation. b Data Mixture

Method

Our method, depicted in Fig. 3, integrates two DMP modules within a CPS-like training framework. The input for our method contain labeled \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^l$$\end{document} , unlabeled \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^u$$\end{document} , and the ground-truth \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{y}^l$$\end{document} . The category-wise weights (in this work, CDifW and DisW) are used for loss calculation and category mask \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\textbf {M}}$$\end{document} generation in the DMP module (see Fig. 4a). For segmentation models \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_B$$\end{document} in the CPS framework, we adopt Exponential Moving Average (EMA) [11] models \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}_B$$\end{document} that have the same structure as the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_B$$\end{document} . The EMA models’ parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{E}^{\hat{f}_A}_t$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{E}^{\hat{f}_B}_t$$\end{document} of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}_B$$\end{document} at iteration t are updated as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{E}^{\hat{f}_A}=\mu \textbf{E}_t^{f_A} + (1-\mu )\textbf{E}^{\hat{f}_A}_{t-1}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{E}^{\hat{f}_B}=\mu \textbf{E}_t^{f_B} + (1-\mu )\textbf{E}^{\hat{f}_B}_{t-1}$$\end{document} during training process, respectively. The DMP modules output mixed labels \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}^m_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}^m_B$$\end{document} , defining new sample pairs for training \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_B$$\end{document} . Framework details are described in Sect. “Double-mix Pseudo-label Framework”.

Distribution based weight (DisW)

We utilized two weights, DisW and CDifW, to estimate the category-wise distribution and difficulty. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^l$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^u$$\end{document} represent the labeled and unlabeled input volumes, respectively.

Following [2], we compute the category-wise distribution weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w^{\textrm{dis}}_{t,k}$$\end{document} for each category k at iteration t during training from the pseudo-labels \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}^u_t$$\end{document} of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^u$$\end{document} , by first calculating the voxel-count ratio \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_{t,k}$$\end{document} , and then normalizing these ratios using the category-wise voxel-counts \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _{t,k}^L$$\end{document} based on the pseudo-labels \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}_t$$\end{document} of the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^l,\textbf{x}^u]$$\end{document} by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w^{\textrm{dis}}_{t,k} & = \frac{\log (r_{t,k})}{\max _{\rho \in \{ 1,...,K\}} \log (r_{t,\rho })},\nonumber \\ r_{t,k} & = \frac{{\max _{\eta \in \{ 1,...,K\}} \psi _{t,\eta }^L}}{\psi ^L_{t,k}}. \end{aligned}$$\end{document}

Weights are updated using an EMA approach

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{W}^{\textrm{dis}}_t = \beta \textbf{W}^{\textrm{dis}}_{t-1} + (1 - \beta )\hat{\textbf{W}}^{\textrm{dis}}_t, \quad \hat{\textbf{W}}^{\textrm{dis}}_t=(w^{\textrm{dis}}_{t,1}, \ldots , w^{\textrm{dis}}_{t,K}), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document} is the parameter for weight smoothing, and K is the number of categories.

Confidence-difficulty based weight (CDifW)

We assess the difficulty of each category through two dimensions: Dice score and confidence.

The learning speed is calculated by population stability index [12] based on the Dice score, which is defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d^u_{t,k}&= \sum _{\varrho =\partial }^{t} \mathbb {I}_{\Delta \le 0 }\ln \left( \frac{\zeta _{\varrho ,k}}{\zeta _{\varrho -1,k}}\right) , \nonumber \\ d^l_{t,k}&= \sum _{\varrho =\partial }^{t} \mathbb {I}_{\Delta > 0}\ln \left( \frac{\zeta _{\varrho ,k}}{\zeta _{\varrho -1,k}}\right) , \nonumber \\ \partial&=\max (t-\tau ,0), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _{t,k}$$\end{document} is the Dice score of the category k at iteration t, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta = \zeta _{t,k}-\zeta _{t-1,k}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {I}_{\Delta \le 0 }$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {I}_{\Delta > 0 }$$\end{document} are defined as the indicator functions,

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {I}_{\Delta \le 0 } = {\left\{ \begin{array}{ll} 1 & \text {if } \Delta \le 0 \\ 0 & \text {if } \Delta> 0 \end{array}\right. }, \quad \mathbb {I}_{\Delta> 0} = {\left\{ \begin{array}{ll} 1 & \text {if } \Delta > 0 \\ 0 & \text {if } \Delta \le 0 \end{array}\right. }. \end{aligned}$$\end{document}

The symbol \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau $$\end{document} represents the cumulative number of iterations, which is empirically set as 50. Following [2], \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d^u_{t,k}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d^l_{t,k}$$\end{document} are used to evaluate whether the category k has been unlearned or well learned, and the difficulty \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{t,k}$$\end{document} should be defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{t,k} = (\frac{d^u_{t,k} + \epsilon }{d^l_{t,k} + \epsilon })^\alpha $$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon $$\end{document} is a smoothing element and the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} is a hyperparameter to alleviate outliers. The well-learned category should perform low \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{t,k}$$\end{document} . The difficulty weight for category k at iteration t is known as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w^{\textrm{dif}}_{t,k} = (1-\zeta _{t,k}) d_{t,k}. \end{aligned}$$\end{document}

As mentioned in the Introduction, it is necessary to utilize confidence in category-wise difficulty. For labeled data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^L$$\end{document} , the confidence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_k$$\end{document} for category k is computed from the logits \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}^L$$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}^L = \text {Softmax}\{f(\textbf{x}^L)\}$$\end{document} and f is the segmentation model applying CDifW, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f\in \{f_A,f_B\}$$\end{document} . At iteration t, the category confidence \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{c}_{t,k}$$\end{document} in a mini-batch is defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{c}_{t,k} = \frac{1}{B}\sum _{b=1}^{B} \frac{1}{z_k}\sum _{j} p^L_{b,k,j}, \end{aligned}$$\end{document}

where B is the mini-batch size and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p^L_{b,k,j}$$\end{document} is the probability of category k at location j in sample b, with j indicating positions marked as category k in the ground-truth. The term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_k$$\end{document} represents the number of pixels for category k in the ground-truth \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{y}$$\end{document} . The EMA method updates the confidence score \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_{t,k}$$\end{document} as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} c_{t,k} = \beta c_{t-1,k} + (1 - \beta ) \hat{c}_{t,k}. \end{aligned}$$\end{document}

Similarly to [13], we define the information score \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{S}_t$$\end{document} for category k at iteration t as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{S}_{t}= & \{\,s_{t,k} \mid k = 1, 2, \dots , K \,\},\nonumber \\ s_{t,k}= & \frac{1-{c_{t,k}}}{\max _{\iota \in \{ 1,...,K\}} (1-c_{t,\iota })}. \end{aligned}$$\end{document}

Then, our proposed CDifW \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{cdif}}$$\end{document} in iteration t can be defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{W}^{\textrm{cdif}}_t=(w^{\textrm{cdif}}_{t,1} ,w^{\textrm{cdif}}_{t,2} \dots , w^{\textrm{cdif}}_{t,k} ), \quad w^{\textrm{cdif}}_{t,k} = s_{t,k}^{\gamma } w^{\textrm{dif}}_{t,k}, \end{aligned}$$\end{document}

where the parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} is a hyperparameter.

This method yields two sets of weights, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{cdif}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{dis}}$$\end{document} , representing training difficulty and category distribution, respectively. We omit subscript t for brevity.

Double-mix pseudo-label module

The scarcity of labeled data necessitates using unlabeled data for augmentation. ClassMix [5] blends regions using pseudo-labels but does not rectify category imbalances, especially in high-difficulty categories, as described in the Introduction. Our DMP module counters this by applying weights \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{cdif}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{dis}}$$\end{document} from Sects. “Distribution based Weight (DisW)” and “Confidence-Difficulty based Weight (CDifW)” to selectively blending categories with ClassMix for more balanced and effective augmentation.

The process of a single DMP module is shown in Fig. 4. Initially, for an input unlabeled volume \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textbf{x}^u $$\end{document} , its pseudo-label \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \hat{\textbf{y}}^u $$\end{document} is computed using the EMA model. For data mixing, a binary mask should be created by the selected categories \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textbf{Z} $$\end{document} . A probability distribution is generated using category-wise weights \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}$$\end{document} (in this work, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W} \in \{\textbf{W}^{\textrm{dis}}, \textbf{W}^{\textrm{cdif}}\}$$\end{document} ), where the weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_k$$\end{document} for category \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k$$\end{document} represents the probability of this category being sampled. We sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k$$\end{document} times from this probability distribution, resulting in a set of selected categories denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{Z}$$\end{document} . Using this categories set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{Z}$$\end{document} , we generate a binary mask \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textbf{M} $$\end{document} corresponding to an unlabeled volume \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textbf{x}^u $$\end{document} as follow: for any given pixel \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ j$$\end{document} in the volume, if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}^u_{j} \in \textbf{Z} $$\end{document} , then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textbf{M}_{j} = 1 $$\end{document} ; otherwise, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \textbf{M}_{j} = 0 $$\end{document} . Therefore, the mixed sample pair \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^m,\hat{\textbf{y}}^m]$$\end{document} using unlabeled data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^u$$\end{document} , labeled data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^l$$\end{document} , and the ground-truth of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^l$$\end{document} can be obtained as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{x}^m = \textbf{x}^u \odot \textbf{M} + \textbf{x}^l \odot (\textbf{1}-\textbf{M}),\quad \hat{\textbf{y}}^m = \hat{\textbf{y}}^u \odot \textbf{M} + \textbf{y}^l \odot (\textbf{1}-\textbf{M}), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\odot $$\end{document} is an element-wise product. As shown in Fig. 3, during the training process, we employ distinct weight distributions to perform two DMP operations to obtain two different mixed sample pairs \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^m_A, \textbf{y}^m_A]$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^m_B, \textbf{y}^m_B]$$\end{document} . This approach considers both the difficulty and distribution of each category, focusing on the imbalanced category augmentation.

Double-mix pseudo-label framework

The process of DMPF is shown in Fig. 3. The updating process of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{dis}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{cdif}}$$\end{document} , as well as the generation process of our proposed DMP, can be summarized in Algorithm 1. To simultaneously consider the distribution and difficulty of categories, we created two models, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_B$$\end{document} , with different random initializations for the model weights. As we defined in Sect. “Distribution based Weight (DisW)”, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N^L$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N^U$$\end{document} show the sample number of the labeled dataset and the unlabeled dataset. We obtained logits \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_B$$\end{document} of the input data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^l,\textbf{x}^u]$$\end{document} through \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_B$$\end{document} . In the CPS framework, the supervised loss is

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L^{s}_{\textrm{seg}}(\textbf{p}_A,\textbf{p}_B,\textbf{y})&= \frac{1}{N^L} \frac{1}{K} \sum _{i=0}^{N^L} [ L_{s}( \textbf{W}^{\textrm{cdif}}, \textbf{p}_{A_i}, \textbf{y}_i) \nonumber \\&\quad + L_{s}(\textbf{W}^{\textrm{dis}}, \textbf{p}_{B_i}, \textbf{y}_i)], \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{y}$$\end{document} is the ground-truth of labeled data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^l$$\end{document} , and for the unsupervised loss component

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L^{s}_u(\textbf{p}_A,\textbf{p}_B, \hat{\textbf{y}}_A, \hat{\textbf{y}}_B)&= \frac{1}{N^L + N^U} \frac{1}{K} \sum _{q=0}^{N^L+N^U} [ L_u(\textbf{W}^{\textrm{cdif}}, \textbf{p}_{A_q}, \hat{\textbf{y}}_{B_q}) \nonumber \\&\quad + L_u(\textbf{W}^{\textrm{dis}}, \textbf{p}_{B_q}, \hat{\textbf{y}}_{A_q})], \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}_A$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}_B$$\end{document} are the pseudo-labels calculated from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_B$$\end{document} . In our experiments, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{s}(\textbf{W}, \textbf{x}, \textbf{y}) = L_{\text {Dice}}( \textbf{W}, \textbf{x}, \textbf{y}) + \frac{1}{2} L_{\text {CE}}(\textbf{W}, \textbf{x}, \textbf{y})$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_u(\textbf{W}, \textbf{x},\textbf{y}) = L_{\text {CE}}( \textbf{W}, \textbf{x}, \textbf{y})$$\end{document} , where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{\text {CE}}$$\end{document} was set as the weighted cross-entropy loss and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L_{\text {Dice}}$$\end{document} was set as the weighted Dice loss [14].

Algorithm 1Double-Mix Pseudo-Label Framework

In the DMP module, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}^u_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{y}}^u_B$$\end{document} are the pseudo-labels generated by the unlabeled volumes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^u$$\end{document} from the EMA model \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ \hat{f}_A$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}_B$$\end{document} . \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^u,\hat{\textbf{y}}^u_A,\textbf{x}^l,\textbf{y}]$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^u,\hat{\textbf{y}}^u_B,\textbf{x}^l,\textbf{y}]$$\end{document} are, respectively, fed into two DMP modules which selecting categories by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{cdif}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{W}^{\textrm{dis}}$$\end{document} . This process is employed to generate new training data pairs at each iteration, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^m_A,\hat{\textbf{y}}^m_A]$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$[\textbf{x}^m_B,\hat{\textbf{y}}^m_B]$$\end{document} . The loss for the data pairs created by the DMP modules is

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L^{m}_{\textrm{seg}}(\textbf{p}_A^m,\textbf{p}_B^m,\hat{\textbf{y}}^m_A,\hat{\textbf{y}}^m_B) = L_{s}(\textbf{W}^{\textrm{cdif}},\textbf{p}_A^m,\hat{\textbf{y}}^m_A) +L_{s} (\textbf{W}^{\textrm{dis}}, \textbf{p}_B^m,\hat{\textbf{y}}^m_B), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_A^m$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}_B^m$$\end{document} are the output of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_A$$\end{document} and model \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_B$$\end{document} with input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}_A^m$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\textbf{x}}_B^m$$\end{document} . Therefore, the loss function can be defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} L & = L^{s}_{\textrm{seg}}(\textbf{p}_A,\textbf{p}_B,\textbf{y}) + L^m_{\textrm{seg}}(\textbf{p}_A^m,\textbf{p}_B^m,\hat{\textbf{y}}^m_A,\hat{\textbf{y}}^m_B) \nonumber \\ & \quad \ + \theta L^{s}_u(\textbf{p}_A, \textbf{p}_B, \hat{\textbf{y}}_A, \hat{\textbf{y}}_B), \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} is a hyperparameters, and the epoch-dependent Gaussian ramp-up strategy [3] is used to enlarge the ratio of unsupervised loss.

In inference stage, for the input volume \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{x}^p$$\end{document} , we calculate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}^p_A = f_A(\textbf{x}^p)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}^p_B = f_B(\textbf{x}^p)$$\end{document} . The predicted logits are given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}^p = \frac{ (\textbf{p}^p_A + \textbf{p}^p_B)}{2}$$\end{document} . The predicted result \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{y}^p$$\end{document} is derived from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{p}^p$$\end{document} by assigning each voxel to the category with the highest predicted probability.Table 1. Segmentation outcomes between our method and other SSL segmentation methods on 40% labeled BTCV datasetMethodsAverage Dice and ASDAverage Dice (%) of Each Class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uparrow $$\end{document} Dice (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uparrow $$\end{document} ASD \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\downarrow $$\end{document} SpRKLKGaEsLiStAoIVCPSVPARAGLAGSS-Net [1] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 42.5 ± 6.549.2 ± 10.184.982.177.110.10.090.251.078.262.00.017.20.00.0DST [18] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 40.1 ± 0.946.8 ± 2.284.978.573.44.50.091.449.972.752.11.012.60.00.0Depl [19] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 41.2 ± 0.948.1 ± 0.582.181.774.23.30.090.851.974.457.10.019.70.00.0CPS [6] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 37.5 ± 2.152.5 ± 11.185.777.971.24.40.0 91.348.166.931.80.010.20.00.0CReST [20] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 38.5 ± 3.822.1 ± 8.762.667.568.918.221.281.742.953.135.76.614.018.19.5CLD [3] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 54.7 ± 1.27.6 ± 0.6** 88.0** 84.6 80.323.028.790.451.579.265.142.824.725.327.0DHC [2] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 59.6 ± 1.24.5 ± 0.6 87.284.179.640.344.883.4 53.6 80.066.646.535.0 33.140.9Ours w/o DMP60.0 ± 0.73.9 ± 0.584.685.580.032.6 45.886.656.279.6 66.8 45.737.528.950.5Ours61.2 ± 0.74.06 ± 0.686.986.0****82.3 35.348.586.553.181.1****71.044.8 35.436.4 48.0The best performance is denoted using bold. The second-best performance is marked by underline. The results are presented in the form of “mean ± std.” \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} We implement methods on our datasetTable 2Segmentation outcomes between our method and other SSL segmentation methods on 5% labeled CHD datasetMethodsAverage Dice and ASDAverage Dice (%) of Each Class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uparrow $$\end{document} Dice (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uparrow $$\end{document} ASD \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\downarrow $$\end{document} LVRVLARAMyAoPASS-Net [1] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 49.7 ± 7.27.9 ± 0.558.448.750.553.359.733.643.5DST [18] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 62.3 ± 1.45.6 ± 0.570.366.168.865.367.248.249.8Depl [19] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 63.6 ± 3.85.1 ± 0.371.966.872.367.966.949.749.6CPS [6] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 62.0 ± 0.95.5 ± 0.371.267.269.263.966.948.347.2CReST [20] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 61.5 ± 1.56.4 ± 0.968.964.671.363.365.845.750.9CLD [3] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 62.4 ± 1.65.9 ± 0.568.664.372.866.366.147.051.5DHC [2] \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${*}$$\end{document} 64.1 ± 0.26.7 ± 0.570.968.472.670.966.049.250.9Ours w/o DMP64.3 ± 1.36.0 ± 0.672.468.972.969.765.449.351.6Ours66.5 ± 0.16.0 ± 0.473.7****72.1****77.368.660.556.9****56.0 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{*}$$\end{document} We implement methods on our dataset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{*}$$\end{document} The best performance is denoted using bold. The second-best performance is marked by underline. The results are presented in the form of "mean ± std"

Fig. 5. Comparative experiments with other methods using 40% BTCV dataset. Some high-difficulty categories are highlighted by red frames Fig. 6. Comparative experiments with other methods using 5% of the CHD dataset

Experiment

Dataset

In this experiment, we validate our approach using two public imbalanced datasets: the BTCV dataset [9] for abdominal organ segmentation, known for its extreme category imbalance, containing 30 annotated cases. For the total cases, 4 were used for validation, 6 for testing, and the others for training. The CHD dataset for Congenital Heart Disease [15] is notable for its categories with balanced voxel-counts and imbalanced segmentation difficulty. The CHD dataset contains 110 CT volumes with labels. Elevan cases were used as the test set, another 11 as the validation set, and the others for the training set. We also tested the effectiveness of DMPF on the balanced dataset, the WRAMC [16], which contains 116 colon cases without annotations, among which we annotated 10 cases, containing the categories of air-area and solid-material, to test the performance of our model on relatively balanced and simpler tasks.

Experimental settings

In our experiment, for the BTCV dataset, we used 10%, 20%, and 40% of the training set as labeled data; for CHD, 5%, 10%, and 20% of the training set as labeled data. For the WRAMC, we applied three-fold cross-validation on ten annotated cases. Unannotated data served as unlabeled training data. We used a patch size of (128, 128, 64) voxels by representing width, height, and depth, respectively. The final segmentation results were obtained using a sliding window strategy with a stride size of (32, 32, 16). We utilized a CPS-like framework with two 5-layer V-Net [10] with kernel numbers of [32, 64, 128, 256, 512] for each layer in the encoder and decoder as the baseline for segmentation. The co-model framework involved two randomly initialized models using the Kaiming normal method [17]. We used random-flip and random-crop to all volumes as the augmentation and used a poly-decay learning strategy [14] with an initial rate of 0.03, optimized via SGD with 0.9 momentum. Hyperparameters were set empirically: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} at 0.5, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document} at 0.99, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma $$\end{document} at 0.2, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} at 0.1. Training employed an early stopping strategy with a 30-epoch threshold. We applied two widely used metrics, Dice score and Average Surface Distance (ASD), in our study.

Results and discussion

Result analysis

In our experiments, we conducted comparisons with several SOTA semi-supervised segmentation methods [1, 6, 18, 19]. Additionally, we adapted category imbalance strategies [2, 3, 20] to the CPS for segmentation tasks, benchmarking them against our approach. We used different initialization seeds for three trials in each experiment. Identical seeds ensure consistent initial parameters across models. “ours w/o DMP” shows the result using the CDifW and DisW without DMP module, while “ours” shows the result using our proposed DMPF.Fig. 7. Results using different percentages of labeled data. a BTCV, b CHDFig. 8The result of Wilcoxon signed-rank test. a The results using 10%, 20%, and 40% of the BTCV dataset as the labeled data, b the results using 5%, 10%, and 20% of the CHD dataset as the labeled data

Figure 1 reveals the BTCV dataset’s imbalanced categories with low voxel-count in Es, RAG, and LAG. As demonstrated in the “ours” of Table 1, our approach improved the Dice score by an average of 4.7% in this imbalanced category than the baseline. Our method also enhanced the Dice score by an average of 5.0% in imbalanced high-difficulty categories RV, Ao, and PA in CHD dataset than the baseline, as shown in Table 2. Figures 5 and 6 show the segmentation results on BTCV and CHD. Our methods performed better than other related methods.Table 3. The results of training CPS module using different weights on 10% labeled BTCV datasetMethodsAverage Dice and ASDDice (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uparrow $$\end{document} ASD \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\downarrow $$\end{document} DHC [2]29.8 ± 5.428.1 ± 8.5CDifW-DisW29.9 ± 2.725.3 ± 8.3CDifW-CDifW26.4 ± 2.825.2 ± 3.5CDifW-CDisW29.4 ± 2.627.3 ± 3.4DisW-DisW26.21 ± 4.732.2 ± 5.0CDisW-CDisW28.9 ± 2.625.5 ± 4.4 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{*}$$\end{document} The best performance is denoted using bold. The second-best performance is marked by underline. The results are presented in the form of "mean ± std"Table 4. Comparison with other data mixture augmentations on 10% labeled BTCV datasetMethodsAverage Dice and ASDDice (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\uparrow $$\end{document} ASD \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\downarrow $$\end{document} CDifW-DisW29.9 ± 2.725.3 ± 8.3CutMix [4]31.5 ± 2.620.4 ± 5.8CutOut [21]30.9 ± 3.724.5 ± 6.8ClassMix [5]29.3 ± 8.333.1 ± 7.0Ours35.7 ± 1.0****18.2 ± 4.3 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{*}$$\end{document} The best performance is denoted using bold. The second-best performance is marked by underline. The results are presented in the form of "mean ± std"

Fig. 9. Dice and ASD results from three-fold cross-validation on colon dataset. a Dice scores, b ASDs Fig. 10. Segmentation results on balanced colon segmentation dataset. The arrow indicates a potential erroneous segmentation likely due to the disruption of spatial information by ClassMix

Furthermore, our proposed CDifW improved the Dice score by an average of 3.0% in the imbalanced categories on BTCV, and 0.4% in the imbalanced categories on CHD, as shown in the “ours w/o DMP” in Tables 1 and 2.

Our method boosted segmentation accuracy in these challenging categories, as evidenced by the average Dice score improvements of 2.3% and 1.5% than the baseline in the 5% CHD and 40% BTCV labeled datasets, respectively. This was achieved through targeted data augmentation, considering category-wise confidence and distribution. The method’s effectiveness on different percentages is shown in Fig. 7. Experimental results indicated significant improvement in segmentation performance considering category-wise confidence. Subsequently, employing the DMP module for data augmentation in high-difficulty categories effectively enhanced overall model performance. Additionally, our method demonstrated more pronounced effectiveness when working with low percentages of labeled data, as shown in Fig. 7.

We conducted Wilcoxon signed-rank tests on the average Dice scores for each sample, comparing the results across different dataset splits and experimental conditions. Figure 8a shows the results using 10%, 20%, and 40% of the BTCV dataset as the labeled data, while Fig. 8b illustrates the results using 5%, 10%, and 20% of the CHD dataset as the labeled data. In all experiments, the Dice scores of our method were statistically higher than those of the second-best method, with all p-values being less than 0.05 ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p < 0.05$$\end{document} ). These results confirm the superiority of our method across different levels of labeled data availability.

Ablation studies

We conducted two ablation studies to examine: (1) the effect of using two distinct category-wise weights in model training, and (2) a comparison with other sample mixture methods.

In Table 3, we present comparative experiments with different weight combinations. For instance, "CDifW-DisW" used CDifW and DisW weights, respectively, during CPS model training. Additionally, CDisW is a weight derived by applying confidence score in Eq. (6) to DisW in Eq. (1), which consequently paid attention to both category-wise difficulty and distribution to a certain extent.

When the weights used in both models were similar (e.g., DisW-DisW, CDifW-CDifW), the models failed to comprehensively consider the category-wise differences in difficulty or distribution, leading to a decline in performance. Meanwhile, CDisW, by considering both category-wise difficulty and distribution, can improve model performance (as in CDisW-CDisW). However, combining CDifW-CDisW reduces heterogeneity due to applying confidence on DistW, causing lower performance. We hypothesized that high heterogeneity weight pairs can increase the model performance, and our proposed CDifW-DisW may provide higher heterogeneity than other methods.

Table 4 presents the results of training our method compared to other data augmentation techniques on 10% of the BTCV dataset. All the methods were based on a CDifW-DisW-applied CPS framework. The results demonstrate that the DMP module, which focuses on category-wise differences for data augmentation, performed best in the imbalanced dataset.

Discussion on balanced dataset

We evaluated our method within a balanced colon segmentation dataset. The DifW was described in Eq. (3). As illustrated in Figs. 9 and 10, our “CDifW-DisW” focusing on category confidence enhanced the segmentation performance for each category. However, a notable limitation is that applying the DMP module (“CDifW-DisW+DMP”) to the balanced dataset likely compromises some spatial information, resulting in reduced performance, as shown in Fig. 10.

Conclusion

This study presents the development and validation of the category-wise CDifW and DMP modules to enhance segmentation performance on imbalanced medical data. The CDifW module considers segmentation difficulty weights based on category-wise confidence, smoothing the weight during training, both improving segmentation in the imbalanced dataset and showing notable benefits in simpler tasks. The DMP module complements CDifW by augmenting the samples of difficult categories using a novel combination of actual and pseudo-labels, which surpasses other data mixture augmentation methods in performance. While our method notably improves performance, especially in high-difficulty categories and imbalanced images, the DMP module introduces segmentation noise in simpler tasks, as illustrated in Fig. 10. Future research will aim to reduce segmentation noise in DMPF by considering marginal contextual information and improve imbalance handling by refining the CDifW.