On Semi-supervised Estimation of Discrete Distributions under f-divergences

TL;DR

This paper investigates the minimax estimation of joint discrete distributions under various f-divergences and loss functions, confirming the optimality of certain estimators for a broad class of divergences and loss metrics.

Contribution

It proves the minimax optimality of univariate estimator combinations for a range of p-values and f-divergences, extending previous results to include p ≤ 2 and multiple divergence measures.

Findings

Establishes minimax optimality for p in [1,2], including l1 loss.

Confirms optimality for divergences like KL, chi-squared, Hellinger, and Le Cam.

Extends prior work to broader divergence and loss function ranges.

Abstract

We study the problem of estimating the joint probability mass function (pmf) over two random variables. In particular, the estimation is based on the observation of $m$ samples containing both variables and $n$ samples missing one fixed variable. We adopt the minimax framework with $l^p_p$ loss functions. Recent work established that univariate minimax estimator combinations achieve minimax risk with the optimal first-order constant for $p \ge 2$ in the regime $m = o(n)$, questions remained for $p \le 2$ and various $f$-divergences. In our study, we affirm that these composite estimators are indeed minimax optimal for $l^p_p$ loss functions, specifically for the range $1 \le p \le 2$, including the critical $l_1$ loss. Additionally, we ascertain their optimality for a suite of $f$-divergences, such as KL, $\chi^2$, Squared Hellinger, and Le Cam divergences.