Do Ensembling and Meta-Learning Improve Outlier Detection in Randomized Controlled Trials?

TL;DR

This study evaluates various machine learning algorithms for outlier detection in large multi-centre randomized controlled trials, introduces a new ensemble method, and compares its performance to existing meta-learning techniques.

Contribution

The paper introduces the Meta-learned Probabilistic Ensemble (MePE), a novel aggregation method for unsupervised outlier detection models, and provides a comprehensive empirical evaluation on real-world clinical trial data.

Findings

Existing algorithms often detect irregularities without supervision.

No single algorithm performs consistently across datasets.

Small ensembles outperform meta-learning approaches on average.

Abstract

Modern multi-centre randomized controlled trials (MCRCTs) collect massive amounts of tabular data, and are monitored intensively for irregularities by humans. We began by empirically evaluating 6 modern machine learning-based outlier detection algorithms on the task of identifying irregular data in 838 datasets from 7 real-world MCRCTs with a total of 77,001 patients from over 44 countries. Our results reinforce key findings from prior work in the outlier detection literature on data from other domains. Existing algorithms often succeed at identifying irregularities without any supervision, with at least one algorithm exhibiting positive performance 70.6% of the time. However, performance across datasets varies substantially with no single algorithm performing consistently well, motivating new techniques for unsupervised model selection or other means of aggregating potentially discordant predictions from multiple candidate models. We propose the Meta-learned Probabilistic Ensemble (MePE), a simple algorithm for aggregating the predictions of multiple unsupervised models, and show that it performs favourably compared to recent meta-learning approaches for outlier detection model selection. While meta-learning shows promise, small ensembles outperform all forms of meta-learning on average, a negative result that may guide the application of current outlier detection approaches in healthcare and other real-world domains.