To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets

TL;DR

This study investigates the performance of ridge logistic regression in small or sparse datasets, highlighting issues with tuning and demonstrating that pre-specifying shrinkage can improve results in such challenging scenarios.

Contribution

It provides a comprehensive simulation analysis comparing ridge logistic regression with Firth's correction, emphasizing the importance of pre-specified shrinkage over tuning in sparse data contexts.

Findings

Tuning ridge regression parameters in small datasets leads to high variability and poor performance.

Pre-specifying the shrinkage parameter yields more accurate coefficients and predictions in sparse settings.

Optimized tuning parameters are negatively correlated with true optimal values in small or sparse datasets.

Abstract

For finite samples with binary outcomes penalized logistic regression such as ridge logistic regression (RR) has the potential of achieving smaller mean squared errors (MSE) of coefficients and predictions than maximum likelihood estimation. There is evidence, however, that RR is sensitive to small or sparse data situations, yielding poor performance in individual datasets. In this paper, we elaborate this issue further by performing a comprehensive simulation study, investigating the performance of RR in comparison to Firth's correction that has been shown to perform well in low-dimensional settings. Performance of RR strongly depends on the choice of complexity parameter that is usually tuned by minimizing some measure of the out-of-sample prediction error or information criterion. Alternatively, it may be determined according to prior assumptions about true effects. As shown in our simulation and illustrated by a data example, values optimized in small or sparse datasets are negatively correlated with optimal values and suffer from substantial variability which translates into large MSE of coefficients and large variability of calibration slopes. In contrast, if the degree of shrinkage is pre-specified, accurate coefficients and predictions can be obtained even in non-ideal settings such as encountered in the context of rare outcomes or sparse predictors.