Investigation of a Data Split Strategy Involving the Time Axis in Adverse Event Prediction Using Machine Learning

TL;DR

This study compares time-based and random data splitting strategies in machine learning models for adverse event prediction, highlighting differences in performance and potential confounding issues, emphasizing the importance of appropriate evaluation methods.

Contribution

The paper provides a comprehensive comparison of time and random data splits in adverse event prediction, revealing the impact on model performance and confounding risks.

Findings

Random split yields higher AUC than time split for most targets.

Chemical space similarity suggests applicability domain alone doesn't explain performance differences.

Knowledge-based information may introduce confounding in time split evaluations.

Abstract

Adverse events are a serious issue in drug development and many prediction methods using machine learning have been developed. The random split cross-validation is the de facto standard for model building and evaluation in machine learning, but care should be taken in adverse event prediction because this approach does not match to the real-world situation. The time split, which uses the time axis, is considered suitable for real-world prediction. However, the differences in model performance obtained using the time and random splits are not clear due to the lack of the comparable studies. To understand the differences, we compared the model performance between the time and random splits using nine types of compound information as input, eight adverse events as targets, and six machine learning algorithms. The random split showed higher area under the curve values than did the time split for six of eight targets. The chemical spaces of the training and test datasets of the time split were similar, suggesting that the concept of applicability domain is insufficient to explain the differences derived from the splitting. The area under the curve differences were smaller for the protein interaction than for the other datasets. Subsequent detailed analyses suggested the danger of confounding in the use of knowledge-based information in the time split. These findings indicate the importance of understanding the differences between the time and random splits in adverse event prediction and strongly suggest that appropriate use of the splitting strategies and interpretation of results are necessary for the real-world prediction of adverse events. We provide analysis code and datasets used in the present study (https://github.com/mizuno-group/AE_prediction).