Estimating divergence functionals and the likelihood ratio by convex risk minimization

TL;DR

This paper introduces convex risk minimization methods for estimating divergence functionals and likelihood ratios, providing simple convex optimization solutions with proven consistency, convergence, and optimal minimax rates under certain conditions.

Contribution

It proposes a novel non-asymptotic variational approach for divergence estimation using convex empirical risk minimization, with theoretical guarantees and practical algorithms.

Findings

Estimators are consistent and converge under mild conditions.

Achieve optimal minimax rates in certain regimes.

Algorithms demonstrate practical viability through simulations.

Abstract

We develop and analyze $M$-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of $f$-divergences, which allows the problem of estimating divergences to be tackled via convex empirical risk optimization. The resulting estimators are simple to implement, requiring only the solution of standard convex programs. We present an analysis of consistency and convergence for these estimators. Given conditions only on the ratios of densities, we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes. We derive an efficient optimization algorithm for computing our estimates, and illustrate their convergence behavior and practical viability by simulations.