Median of Forests for Robust Density Estimation

TL;DR

This paper introduces MFRDE, a robust density estimation method using median of forests, which effectively resists all outliers and outperforms existing kernel-based methods in theory and practice.

Contribution

The paper proposes MFRDE, a novel ensemble learning algorithm that enhances robustness in density estimation by using median operations on forest estimators, allowing larger subsampling and better outlier resistance.

Findings

MFRDE achieves near-uncontaminated convergence rates even with many outliers.

It outperforms existing robust kernel methods in real data experiments.

MFRDE is effective in anomaly detection applications.

Abstract

Robust density estimation refers to the consistent estimation of the density function even when the data is contaminated by outliers. We find that existing forest density estimation at a certain point is inherently resistant to the outliers outside the cells containing the point, which we call \textit{non-local outliers}, but not resistant to the rest \textit{local outliers}. To achieve robustness against all outliers, we propose an ensemble learning algorithm called \textit{medians of forests for robust density estimation} (\textit{MFRDE}), which adopts a pointwise median operation on forest density estimators fitted on subsampled datasets. Compared to existing robust kernel-based methods, MFRDE enables us to choose larger subsampling sizes, sacrificing less accuracy for density estimation while achieving robustness. On the theoretical side, we introduce the local outlier exponent to quantify the number of local outliers. Under this exponent, we show that even if the number of outliers reaches a certain polynomial order in the sample size, MFRDE is able to achieve almost the same convergence rate as the same algorithm on uncontaminated data, whereas robust kernel-based methods fail. On the practical side, real data experiments show that MFRDE outperforms existing robust kernel-based methods. Moreover, we apply MFRDE to anomaly detection to showcase a further application.