Learning Coefficients in Semi-Regular Models

TL;DR

This paper introduces a novel method using algebraic geometry techniques to calculate the learning coefficients in semi-regular models, extending understanding beyond regular models and providing exact values for specific singular models.

Contribution

It proposes a new approach leveraging properties of the log-likelihood ratio for calculating the real log canonical threshold in semi-regular models, which was previously elusive.

Findings

Method successfully computes learning coefficients for semi-regular models.

Linear independence in the log-likelihood ratio influences the threshold.

Examples include two-parameter semi-regular models and mixture models.

Abstract

Recent advances have clarified theoretical learning accuracy in Bayesian inference, revealing that the asymptotic behavior of metrics such as generalization loss and free energy, assessing predictive accuracy, is dictated by a rational number unique to each statistical model, termed the learning coefficient (real log canonical threshold) . For models meeting regularity conditions, their learning coefficients are known. However, for singular models not meeting these conditions, exact values of learning coefficients are provided for specific models like reduced-rank regression, but a broadly applicable calculation method for these learning coefficients in singular models remains elusive. The problem of determining learning coefficients relates to finding normal crossings of Kullback-Leibler divergence in algebraic geometry. In this context, it is crucial to perform appropriate coordinate transformations and blow-ups. This paper introduces an approach that utilizes properties of the log-likelihood ratio function for constructing specific variable transformations and blow-ups to uniformly calculate the real log canonical threshold. It was found that linear independence in the differential structure of the log-likelihood ratio function significantly influences the real log canonical threshold. This approach has not been considered in previous research. In this approach, the paper presents cases and methods for calculating the exact values of learning coefficients in statistical models that satisfy a simple condition next to the regularity conditions (semi-regular models), offering examples of learning coefficients for two-parameter semi-regular models and mixture distribution models with a constant mixing ratio.