Sobolev Space Regularised Pre Density Models

TL;DR

This paper introduces a Sobolev norm regularized non-parametric density estimation method that is statistically consistent, interpretable, and effective for anomaly detection, using sampling-based kernel approximation and natural gradient optimization.

Contribution

It presents a novel Sobolev space regularization approach for density estimation, with sampling-based kernel approximation and adapted Fisher divergence score matching.

Findings

Achieves second place on ADBench anomaly detection benchmark.

Demonstrates statistical consistency and interpretability of the model.

Shows effective optimization with natural gradients despite non-convexity.

Abstract

We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive bias of the model clear and interpretable. While there is no closed analytic form for the associated kernel, we show that one can approximate it using sampling. The optimization problem needed to determine the density is non-convex, and standard gradient methods do not perform well. However, we show that with an appropriate initialization and using natural gradients, one can obtain well performing solutions. Finally, while the approach provides pre-densities (i.e. not necessarily integrating to 1), which prevents the use of log-likelihood for cross validation, we show that one can instead adapt Fisher divergence based score matching methods for this task. We evaluate the resulting method on the comprehensive recent anomaly detection benchmark suite, ADBench, and find that it ranks second best, among more than 15 algorithms.