Condition Number Analysis of Kernel-based Density Ratio Estimation

TL;DR

This paper analyzes the numerical stability of kernel-based density ratio estimation methods, revealing that the kernel least-squares approach has superior condition numbers, leading to better convergence and stability.

Contribution

It provides a theoretical condition number analysis of kernel density ratio estimators, highlighting the advantages of the kernel least-squares method over other approaches.

Findings

Kernel least-squares method has a smaller condition number than kernel mean matching.

An alternative formulation of the kernel least-squares estimator has an even smaller condition number.

Numerical experiments confirm the theoretical analysis.

Abstract

The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), and feature selection (mutual information). Recently, several methods of directly estimating the density ratio have been developed, e.g., kernel mean matching, maximum likelihood density ratio estimation, and least-squares density ratio fitting. In this paper, we consider a kernelized variant of the least-squares method and investigate its theoretical properties from the viewpoint of the condition number using smoothed analysis techniques--the condition number of the Hessian matrix determines the convergence rate of optimization and the numerical stability. We show that the kernel least-squares method has a smaller condition number than a version of kernel mean matching and other M-estimators, implying that the kernel least-squares method has preferable numerical properties. We further give an alternative formulation of the kernel least-squares estimator which is shown to possess an even smaller condition number. We show that numerical studies meet our theoretical analysis.