How and why does deep ensemble coupled with transfer learning increase performance in bipolar disorder and schizophrenia classification?

TL;DR

This study investigates how deep ensemble learning combined with transfer learning enhances classification accuracy and stability in bipolar disorder and schizophrenia, revealing optimal ensemble sizes and the impact of pre-training on generalization.

Contribution

It provides insights into the mechanisms by which DE and TL improve model stability and performance, including ensemble size effects and the influence of pre-training on model convergence.

Findings

Deep ensemble reaches performance plateau at 10 models.

Transfer learning constrains models to similar loss basins, improving generalization.

Using TL with DE significantly enhances classifier stability and accuracy.

Abstract

Transfer learning (TL) and deep ensemble learning (DE) have recently been shown to outperform simple machine learning in classifying psychiatric disorders. However, there is still a lack of understanding as to why that is. This paper aims to understand how and why DE and TL reduce the variability of single-subject classification models in bipolar disorder (BD) and schizophrenia (SCZ). To this end, we investigated the training stability of TL and DE models. For the two classification tasks under consideration, we compared the results of multiple trainings with the same backbone but with different initializations. In this way, we take into account the epistemic uncertainty associated with the uncertainty in the estimation of the model parameters. It has been shown that the performance of classifiers can be significantly improved by using TL with DE. Based on these results, we investigate i) how many models are needed to benefit from the performance improvement of DE when classifying BD and SCZ from healthy controls, and ii) how TL induces better generalization, with and without DE. In the first case, we show that DE reaches a plateau when 10 models are included in the ensemble. In the second case, we find that using a pre-trained model constrains TL models with the same pre-training to stay in the same basin of the loss function. This is not the case for DL models with randomly initialized weights.