Low-rank Characteristic Tensor Density Estimation Part II: Compression and Latent Density Estimation

TL;DR

This paper introduces a novel framework combining nonlinear auto-encoders and low-rank tensor models for efficient density estimation in reduced dimensions, improving generative modeling and anomaly detection.

Contribution

It presents a joint optimization method for auto-encoders and a non-parametric latent density estimator using low-rank tensor models, which is more flexible than traditional variational auto-encoders.

Findings

Achieves promising results on toy, tabular, and image datasets.

Effective for regression, sampling, and anomaly detection tasks.

Outperforms some existing methods in density estimation accuracy.

Abstract

Learning generative probabilistic models is a core problem in machine learning, which presents significant challenges due to the curse of dimensionality. This paper proposes a joint dimensionality reduction and non-parametric density estimation framework, using a novel estimator that can explicitly capture the underlying distribution of appropriate reduced-dimension representations of the input data. The idea is to jointly design a nonlinear dimensionality reducing auto-encoder to model the training data in terms of a parsimonious set of latent random variables, and learn a canonical low-rank tensor model of the joint distribution of the latent variables in the Fourier domain. The proposed latent density model is non-parametric and universal, as opposed to the predefined prior that is assumed in variational auto-encoders. Joint optimization of the auto-encoder and the latent density estimator is pursued via a formulation which learns both by minimizing a combination of the negative log-likelihood in the latent domain and the auto-encoder reconstruction loss. We demonstrate that the proposed model achieves very promising results on toy, tabular, and image datasets on regression tasks, sampling, and anomaly detection.