Frequency of Frequencies Distributions and Size Dependent Exchangeable Random Partitions

TL;DR

This paper introduces a flexible probabilistic model for the frequency of frequencies distribution that depends on population size, enabling better modeling of data with size-dependent clustering, demonstrated on real datasets.

Contribution

It proposes a novel cluster structure based on completely random measures, allowing size-dependent exchangeable partitions and FoF modeling, with a new Gibbs sampling method for inference.

Findings

Model effectively captures size-dependent clustering patterns.

Demonstrates improved fit on real text, genomic, and survey data.

Provides a practical Gibbs sampling algorithm for FoF extrapolation.

Abstract

Motivated by the fundamental problem of modeling the frequency of frequencies (FoF) distribution, this paper introduces the concept of a cluster structure to define a probability function that governs the joint distribution of a random count and its exchangeable random partitions. A cluster structure, naturally arising from a completely random measure mixed Poisson process, allows the probability distribution of the random partitions of a subset of a population to be dependent on the population size, a distinct and motivated feature that makes it more flexible than a partition structure. This allows it to model an entire FoF distribution whose structural properties change as the population size varies. A FoF vector can be simulated by drawing an infinite number of Poisson random variables, or by a stick-breaking construction with a finite random number of steps. A generalized negative binomial process model is proposed to generate a cluster structure, where in the prior the number of clusters is finite and Poisson distributed, and the cluster sizes follow a truncated negative binomial distribution. We propose a simple Gibbs sampling algorithm to extrapolate the FoF vector of a population given the FoF vector of a sample taken without replacement from the population. We illustrate our results and demonstrate the advantages of the proposed models through the analysis of real text, genomic, and survey data.