Adaptive Kernel Density Estimation with Pre-training

TL;DR

This paper introduces a pre-training approach for kernel density estimation that leverages neural networks to recommend adaptive kernels, significantly improving accuracy in high-dimensional settings.

Contribution

It applies pre-training to non-parametric density estimation, enabling neural networks to suggest adaptive kernels for each sample, enhancing efficiency and accuracy.

Findings

Pre-trained neural networks effectively recommend adaptive kernels in high-dimensional density estimation.

The method improves accuracy when the target distribution is similar to the pre-training distribution.

Fine-tuning can recover benefits when the target distribution differs substantially from the pre-training distribution.

Abstract

Density estimation in high-dimensional settings is an important and challenging statistical problem.Traditional methods based on kernel smoothing are inefficient in high dimensions due to the difficulties in specifying appropriate location-adaptive kernels. In this work, we introduce pre-training, a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network that can recommend an appropriate location-adaptive kernel for each sample point, efficient density estimation with adaptive kernels is achieved in high dimensions. A wide range of numerical experiments show that this strategy is highly effective for improving density-estimation accuracy, when the target distribution is close to the distribution family for pre-training. When the target distribution is substantially different from the pre-training distribution family, the benefit from the proposed pre-training strategy may be diluted, but can be reactivated by an additional fine-tuning procedure.