Bounds on Lp errors in density ratio estimation via f-divergence loss functions

TL;DR

This paper derives theoretical bounds on the $L_p$ errors in density ratio estimation using $f$-divergence loss functions, revealing how errors grow with divergence and data dimension, and validates these bounds with experiments.

Contribution

It provides the first general bounds on $L_p$ errors for density ratio estimators based on $f$-divergence, applicable to a broad class of Lipschitz estimators.

Findings

Lower bounds depend exponentially on KL divergence for $p > 1$

Error bounds scale with data dimensionality and expected density ratio

Numerical experiments confirm theoretical bounds

Abstract

Density ratio estimation (DRE) is a core technique in machine learning used to capture relationships between two probability distributions. $f$-divergence loss functions, which are derived from variational representations of $f$-divergence, have become a standard choice in DRE for achieving cutting-edge performance. This study provides novel theoretical insights into DRE by deriving upper and lower bounds on the $L_p$ errors through $f$-divergence loss functions. These bounds apply to any estimator belonging to a class of Lipschitz continuous estimators, irrespective of the specific $f$-divergence loss function employed. The derived bounds are expressed as a product involving the data dimensionality and the expected value of the density ratio raised to the $p$-th power. Notably, the lower bound includes an exponential term that depends on the Kullback--Leibler (KL) divergence, revealing that the $L_p$ error increases significantly as the KL divergence grows when $p > 1$. This increase becomes even more pronounced as the value of $p$ grows. The theoretical insights are validated through numerical experiments.