A Spectral Framework for Closed-Form Relative Density Estimation

TL;DR

This paper introduces a spectral framework for efficiently estimating relative log-densities in probabilistic models using closed-form solutions based on spectral analysis and moments.

Contribution

The authors develop a novel spectral approach that provides explicit closed-form estimators for divergences and log-densities, extending to various f-divergences and neural network features.

Findings

Derived explicit spectral formulas using first- and second-order moments.

Proved convergence guarantees for the estimators.

Empirically validated the framework on synthetic data and compared with variational methods.

Abstract

We propose a closed-form spectral framework for relative log-density estimation in linearly parameterized probabilistic models, including unnormalized and conditional models. This is achieved by representing the Kullback-Leibler (KL) divergence as an integral of weighted chi-squared divergences, converting KL estimation into a family of least-squares problems. We derive an explicit spectral formula based only on first- and second-order feature moments, yielding closed-form estimators of both divergences and log-density potentials for fixed features. The framework extends to a broad class of f-divergences and can be combined with kernelization or feature learning with neural networks. We prove convergence guarantees for the resulting estimators and empirically compare them on synthetic data with optimization-based variational formulations, including logistic and softmax regression for normalized conditional models.