Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

TL;DR

This paper proves that in singular learning models, Bayes cross-validation loss and the widely applicable information criterion are asymptotically equivalent, linking their behavior to algebraic geometric properties of the model.

Contribution

It establishes the asymptotic equivalence of Bayes cross-validation and the widely applicable information criterion in singular models, extending classical results to more complex models.

Findings

Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion.

The sum of generalization and cross-validation errors relates to the real log canonical threshold.

Deviance information criteria differ from the studied criteria in singular models.

Abstract

In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to $2\lambda/n$, where $\lambda$ is the real log canonical threshold and $n$ is the number of training samples. Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion.