Statistical Learning Theory of Quasi-Regular Cases

TL;DR

This paper introduces the concept of quasi-regular cases in statistical learning, where singular models exhibit properties similar to regular models, allowing explicit calculation of key invariants and understanding of errors.

Contribution

It defines quasi-regular cases, proves the equality of birational invariants in these cases, and provides explicit values for these invariants, advancing the understanding of singular learning models.

Findings

Birational invariants are equal in quasi-regular cases.

Explicit formulas for invariants are derived.

Symmetry between generalization and training errors is established.

Abstract

Many learning machines such as normal mixtures and layered neural networks are not regular but singular statistical models, because the map from a parameter to a probability distribution is not one-to-one. The conventional statistical asymptotic theory can not be applied to such learning machines because the likelihood function can not be approximated by any normal distribution. Recently, new statistical theory has been established based on algebraic geometry and it was clarified that the generalization and training errors are determined by two birational invariants, the real log canonical threshold and the singular fluctuation. However, their concrete values are left unknown. In the present paper, we propose a new concept, a quasi-regular case in statistical learning theory. A quasi-regular case is not a regular case but a singular case, however, it has the same property as a regular case. In fact, we prove that, in a quasi-regular case, two birational invariants are equal to each other, resulting that the symmetry of the generalization and training errors holds. Moreover, the concrete values of two birational invariants are explicitly obtained, the quasi-regular case is useful to study statistical learning theory.