Comparing weighted difference and earth mover's distance via Young diagrams

TL;DR

This paper investigates the relationship between two histogram distance metrics, weighted difference and earth mover's distance, deriving formulas for their probability and expectation using Young diagrams, and extends the analysis to multiple histograms.

Contribution

It introduces a combinatorial approach using Young diagrams to analyze the probability and expected value of the earth mover's distance equaling the weighted difference, and generalizes to multiple histograms.

Findings

Derived a formula for the probability that EMD equals the absolute difference.

Calculated the expected value of the weighted difference between histograms.

Extended the analysis to multiple histograms using root lattice geometry.

Abstract

We consider two natural statistics on pairs of histograms, in which the $n$ bins have weights $0, \ldots, n-1$. The difference ($\mathbf{D}$) between the weighted totals of the histograms is, in a sense, refined by the earth mover's distance ($\mathbf{EMD}$), which measures the amount of work required to equalize the histograms. We were recently surprised, however, by how little $\mathbf{EMD}$ actually does refine $\mathbf{D}$ in certain real-world applications, which led to the main problem in this paper: what is the probability that $\mathbf{EMD} = |\mathbf{D}|$? We derive a formula for this probability, as well as the expected value of $|\mathbf{D}|$, via the combinatorics of Young diagrams and plane partitions. We then generalize our results to an arbitrary number of histograms, where we realize this higher-dimensional $\mathbf{D}$ as distance on the Type-A root lattice.