The Geometry of Statistical Data and Information: A Large Deviation Perspective

TL;DR

This paper explores the geometric structure of empirical data spaces using large deviation theory, linking information geometry with probability foundations beyond traditional i.i.d. assumptions.

Contribution

It introduces a large deviation perspective to study data geometry, connecting information projection with Kolmogorov's probability theory under various assumptions.

Findings

Entropy functions depend on the measure and data, shaping the data space geometry.

The Fisher-Rao metric's spherical geometry applies only under i.i.d. assumptions.

Information projection aligns with Kolmogorov's probability theory, even beyond i.i.d. cases.

Abstract

The manifold of empirical mean values of statistical data ad infinitum has a geometric shape that depends on the probability measure that governs the generating model. Large deviation theory produces entropy functions that depend on both the probability measure and the statistical data; we use entropy to study the geometry of the data space rather than that of the space of probability distributions. It is well known, since Rao's work, that the Fisher-Rao metric makes the probability simplex into a sphere. From our perspective, that result translates to the space of empirical singleton counting frequencies under an i.i.d. assumption. Following our ideas and going beyond i.i.d., the choice of measure curves the space. When we study the pairwise statistics, the spherical geometry breaks down entirely. We show that the information projection, defined in information geometry as divergence minimization, coincides with the information projection in Kolmogorov's probability theory. This identification holds under both i.i.d. and Markovian assumptions and connects information geometry to the foundations of probability theory.