Semi-Supervised Approaches to Efficient Evaluation of Model Prediction Performance

TL;DR

This paper introduces a semi-supervised method for efficiently evaluating the prediction performance of regression models, especially in medical applications where labeled data is costly, demonstrating improved accuracy over supervised methods.

Contribution

It proposes a novel two-step semi-supervised estimation procedure for model evaluation that reduces variance and corrects for overfitting, applicable to phenotyping algorithms from electronic medical records.

Findings

Semi-supervised estimators are consistent and asymptotically normal.

SS estimators have smaller variance than supervised ones under correct model specification.

Method performs well in simulations and real EMR studies.

Abstract

In many modern machine learning applications, the outcome is expensive or time-consuming to collect while the predictor information is easy to obtain. Semi-supervised learning (SSL) aims at utilizing large amounts of `unlabeled' data along with small amounts of `labeled' data to improve the efficiency of a classical supervised approach. Though numerous SSL classification and prediction procedures have been proposed in recent years, no methods currently exist to evaluate the prediction performance of a working regression model. In the context of developing phenotyping algorithms derived from electronic medical records (EMR), we present an efficient two-step estimation procedure for evaluating a binary classifier based on various prediction performance measures in the semi-supervised (SS) setting. In step I, the labeled data is used to obtain a non-parametrically calibrated estimate of the conditional risk function. In step II, SS estimates of the prediction accuracy parameters are constructed based on the estimated conditional risk function and the unlabeled data. We demonstrate that under mild regularity conditions the proposed estimators are consistent and asymptotically normal. Importantly, the asymptotic variance of the SS estimators is always smaller than that of the supervised counterparts under correct model specification. We also correct for potential overfitting bias in the SS estimators in finite sample with cross-validation and develop a perturbation resampling procedure to approximate their distributions. Our proposals are evaluated through extensive simulation studies and illustrated with two real EMR studies aiming to develop phenotyping algorithms for rheumatoid arthritis and multiple sclerosis.