Accelerating Cross-Validation in Multinomial Logistic Regression with $\ell_1$-Regularization

TL;DR

This paper introduces an approximate formula to efficiently evaluate cross-validation estimates for multinomial logistic regression with $\, ext{l}_1$-regularization, significantly reducing computational costs.

Contribution

The authors develop a novel perturbative approximation method for cross-validation in regularized multinomial logistic regression, extending it to elastic net regularization.

Findings

The approximate formula accurately estimates cross-validation likelihoods.

Significant reduction in computational time compared to traditional methods.

Validated on simulated and real datasets, including ISOLET.

Abstract

We develop an approximate formula for evaluating a cross-validation estimator of predictive likelihood for multinomial logistic regression regularized by an $\ell_1$-norm. This allows us to avoid repeated optimizations required for literally conducting cross-validation; hence, the computational time can be significantly reduced. The formula is derived through a perturbative approach employing the largeness of the data size and the model dimensionality. An extension to the elastic net regularization is also addressed. The usefulness of the approximate formula is demonstrated on simulated data and the ISOLET dataset from the UCI machine learning repository.