Can we Improve Prediction of Psychotherapy Outcomes Through Pretraining With Simulated Data?

TL;DR

This study investigates whether pretraining machine learning models with simulated data improves psychotherapy outcome predictions, finding mixed results and highlighting challenges like data scarcity.

Contribution

It introduces a novel approach of using literature-based simulated data for pretraining, then fine-tuning on real data, and evaluates its effectiveness.

Findings

Pretraining showed some descriptive improvements but no significant advantage.

In the second study, models trained only on real data outperformed pretrained ones.

Challenges include limited informative publications and data scarcity.

Abstract

In the context of personalized medicine, machine learning algorithms are growing in popularity. These algorithms require substantial information, which can be acquired effectively through the usage of previously gathered data. Open data and the utilization of synthetization techniques have been proposed to address this. In this paper, we propose and evaluate alternative approach that uses additional simulated data based on summary statistics published in the literature. The simulated data are used to pretrain random forests, which are afterwards fine-tuned on a real dataset. We compare the predictive performance of the new approach to random forests trained only on the real data. A Monte Carlo Cross Validation (MCCV) framework with 100 iterations was employed to investigate significance and stability of the results. Since a first study yielded inconclusive results, a second study with improved methodology (i.e., systematic information extraction and different prediction outcome) was conducted. In Study 1, some pretrained random forests descriptively outperformed the standard random forest. However, this improvement was not significant (t(99) = 0.89, p = 0.19). Contrary to expectations, in Study 2 the random forest trained only with the real data outperformed the pretrained random forests. We conclude with a discussion of challenges, such as the scarcity of informative publications, and recommendations for future research.