Fitting Semiparametric Cumulative Probability Models for Big Data

TL;DR

This paper introduces three scalable methods for fitting cumulative probability models to large datasets, improving computational efficiency while maintaining accuracy, demonstrated through simulations and a large-scale application.

Contribution

It proposes divide-and-combine, binning, and rounding approaches to make CPMs feasible for big data, with theoretical consistency and practical performance evaluation.

Findings

Methods perform well in simulations

Parameter estimates are consistent

Approaches reduce running time and memory usage

Abstract

Cumulative probability models (CPMs) are a robust alternative to linear models for continuous outcomes. However, they are not feasible for very large datasets due to elevated running time and memory usage, which depend on the sample size, the number of predictors, and the number of distinct outcomes. We describe three approaches to address this problem. In the divide-and-combine approach, we divide the data into subsets, fit a CPM to each subset, and then aggregate the information. In the binning and rounding approaches, the outcome variable is redefined to have a greatly reduced number of distinct values. We consider rounding to a decimal place and rounding to significant digits, both with a refinement step to help achieve the desired number of distinct outcomes. We show with simulations that these approaches perform well and their parameter estimates are consistent. We investigate how running time and peak memory usage are influenced by the sample size, the number of distinct outcomes, and the number of predictors. As an illustration, we apply the approaches to a large publicly available dataset investigating matrix multiplication runtime with nearly one million observations.