Fast leave-one-cluster-out cross-validation using clustered Network Information Criterion (NICc)

TL;DR

This paper introduces a fast clustered estimator of NICc for model validation on clustered data, improving accuracy over AIC and BIC in model selection, especially with strong clustering.

Contribution

The paper develops NICc, a new cluster-adjusted criterion that approximates leave-one-cluster-out deviance efficiently, enhancing model validation for clustered datasets.

Findings

NICc provides a more accurate approximation than AIC and BIC.

NICc improves variable selection accuracy in clustered data.

Simulation and empirical results show NICc's effectiveness with strong clustering.

Abstract

For prediction models developed on clustered data that do not account for cluster heterogeneity in model parameterization, it is crucial to use cluster-based validation to assess model generalizability on unseen clusters. This paper introduces a clustered estimator of the Network Information Criterion (NICc) to approximate leave-one-cluster-out deviance for standard prediction models with twice differentiable log-likelihood functions. The NICc serves as a fast alternative to cluster-based cross-validation. Stone (1977) proved that the Akaike Information Criterion (AIC) is asymptotically equivalent to leave-one-observation-out cross-validation for true parametric models with independent and identically distributed observations. Ripley (1996) noted that the Network Information Criterion (NIC), derived from Stone's proof, is a better approximation when the model is misspecified. For clustered data, we derived NICc by substituting the Fisher information matrix in the NIC with a clustering-adjusted estimator. The NICc imposes a greater penalty when the data exhibits stronger clustering, thereby allowing the NICc to better prevent over-parameterization. In a simulation study and an empirical example, we used standard regression to develop prediction models for clustered data with Gaussian or binomial responses. Compared to the commonly used AIC and BIC for standard regression, NICc provides a much more accurate approximation to leave-one-cluster-out deviance and results in more accurate model size and variable selection, as determined by cluster-based cross-validation, especially when the data exhibit strong clustering.