Sometimes nonparametrics beat parametrics, even when the model is right

TL;DR

This paper demonstrates that nonparametric density estimators, like kernel density estimators, can outperform parametric methods such as maximum likelihood estimation in small samples, challenging conventional assumptions about model correctness.

Contribution

It provides a counter-intuitive example showing nonparametric methods can outperform parametric ones even when the parametric model is correct, especially in small sample scenarios.

Findings

Kernel density estimators can outperform parametric estimators in small samples.

Nonparametric methods are not always inferior to parametric methods when models are correct.

Small-sample analysis reveals surprising performance advantages of nonparametric estimators.

Abstract

A basic issue in both teaching of and practice of statistics is the interplay between modelling assumptions and inference performance. The general message conveyed is that stronger assumptions lead to better statistical performance of the relevant estimators, tests and confidence intervals, provided that these assumptions hold. On the other hand, fewer assumptions often lead to safer and more robust methods that are good also outside narrow conditions, but not quite as good as specialist methods that exploit such narrower conditions, if these are fulfilled. This interplay is nicely illustrated in the context of density estimation, where parametric and nonparametric methods can be contrasted. The parametric ones have mean squared errors of size $O(n^{-1})$ in terms of sample size $n$ if the parametric model is right, but are not even consistent outside the model. The nonparametric methods are everywhere consistent and have mean squared errors of size $O(n^{-4/5})$ for broad classes of estimands. The point we are making here is that this picture is not universally true! We show that a simple kernel density estimator can perform better than a directly estimated parametric density on the latter's home turf, for small sample sizes, in the sense of mean integrated squared error. Our main example is that of estimating an unknown normal density. In the process of developing and discussing this somewhat counter-intuitive and half-paradoxical example we touch on several tangential issues of interest, pertaining to exact small-sample analysis of density estimators.