Asymptotic Behavior of Free Energy When Optimal Probability Distribution Is Not Unique

TL;DR

This paper investigates the asymptotic behavior of free energy and generalization loss in Bayesian inference models with multiple optimal probability distributions, revealing new terms in their asymptotic analysis.

Contribution

It provides a theoretical derivation of the asymptotic behaviors of free energy and generalization loss when the optimal distribution is not unique, extending conventional analysis.

Findings

Asymptotic behaviors include new terms not seen in traditional models.

The analysis applies to singular models with multiple optima.

Results improve understanding of Bayesian inference in complex models.

Abstract

Bayesian inference is a widely used statistical method. The free energy and generalization loss, which are used to estimate the accuracy of Bayesian inference, are known to be small in singular models that do not have a unique optimal parameter. However, their characteristics are not yet known when there are multiple optimal probability distributions. In this paper, we theoretically derive the asymptotic behaviors of the generalization loss and free energy in the case that the optimal probability distributions are not unique and show that they contain asymptotically different terms from those of the conventional asymptotic analysis.