A Theory of Taxonomy

TL;DR

This paper introduces a universal branching model that explains the distribution of items across categories in large taxonomies, with applications spanning ecology, computer science, and library sciences.

Contribution

It proposes a simple, non-parametric model that reproduces observed abundance distributions in diverse real-world datasets, revealing underlying commonalities.

Findings

The model accurately fits data from NYC transit, libraries, and microbiomes.

It predicts unrepresented categories in finite samples.

A universal pattern in taxonomic abundance distributions is identified.

Abstract

A taxonomy is a standardized framework to classify and organize items into categories. Hierarchical taxonomies are ubiquitous, ranging from the classification of organisms to the file system on a computer. Characterizing the typical distribution of items within taxonomic categories is an important question with applications in many disciplines. Ecologists have long sought to account for the patterns observed in species-abundance distributions (the number of individuals per species found in some sample), and computer scientists study the distribution of files per directory. Is there a universal statistical distribution describing how many items are typically found in each category in large taxonomies? Here, we analyze a wide array of large, real-world datasets -- including items lost and found on the New York City transit system, library books, and a bacterial microbiome -- and discover such an underlying commonality. A simple, non-parametric branching model that randomly categorizes items and takes as input only the total number of items and the total number of categories successfully reproduces the abundance distributions in these datasets. This result may shed light on patterns in species-abundance distributions long observed in ecology. The model also predicts the number of taxonomic categories that remain unrepresented in a finite sample.