Information theory with finite vector spaces

TL;DR

This paper explores the connection between quadratic (Tsallis 2-) entropy and finite vector spaces, providing combinatorial, probabilistic, and information-theoretic insights into nonadditivity and applications to source coding.

Contribution

It establishes a link between quadratic entropy and $q$-multinomial coefficients, introducing a stochastic process for vector space configurations and extending the asymptotic equipartition property.

Findings

$q$-multinomial coefficients count flags of finite vector spaces.

The quadratic entropy explains the size of typical subspaces.

Applications to source coding with vector space messages.

Abstract

Whereas Shannon entropy is related to the growth rate of multinomial coefficients, we show that the quadratic entropy (Tsallis 2-entropy) is connected to their $q$-deformation; when $q$ is a prime power, these $q$-multinomial coefficients count flags of finite vector spaces with prescribed length and dimensions. In particular, the $q$-binomial coefficients count vector subspaces of given dimension. We obtain this way a combinatorial explanation for the nonadditivity of the quadratic entropy, which arises from a recursive counting of flags. We show that statistical systems whose configurations are described by flags provide a frequentist justification for the maximum entropy principle with Tsallis statistics. We introduce then a discrete-time stochastic process associated to the $q$-binomial probability distribution, that generates at time $n$ a vector subspace of $\mathbb{F}_q^n$ (here $\mathbb{F}_q$ is the finite field of order $q$). The concentration of measure on certain "typical subspaces" allows us to extend the asymptotic equipartition property to this setting. The size of the typical set is quantified by the quadratic entropy. We discuss the applications to Shannon theory, particularly to source coding, when messages correspond to vector spaces.