Existence of Direct Density Ratio Estimators

TL;DR

This paper characterizes when the KLIEP density ratio estimator exists, linking it to the convex hull of sufficient statistics, and discusses implications for high-dimensional regularized estimation in differential network analysis.

Contribution

It provides a theoretical characterization of the existence conditions for KLIEP estimators, including regularized versions, in high-dimensional settings.

Findings

Existence of KLIEP depends on average sufficient statistic's position relative to convex hull.

Regularized KLIEP requires tuning parameter to exceed a dual norm-based distance.

Implications for differential network analysis are discussed.

Abstract

Many two-sample problems call for a comparison of two distributions from an exponential family. Density ratio estimation methods provide ways to solve such problems through direct estimation of the differences in natural parameters. The term direct indicates that one avoids estimating both marginal distributions. In this context, we consider the Kullback--Leibler Importance Estimation Procedure (KLIEP), which has been the subject of recent work on differential networks. Our main result shows that the existence of the KLIEP estimator is characterized by whether the average sufficient statistic for one sample belongs to the convex hull of the set of all sufficient statistics for data points in the second sample. For high-dimensional problems it is customary to regularize the KLIEP loss by adding the product of a tuning parameter and a norm of the vector of parameter differences. We show that the existence of the regularized KLIEP estimator requires the tuning parameter to be no less than the dual norm-based distance between the average sufficient statistic and the convex hull. The implications of these existence issues are explored in applications to differential network analysis.