Data-adaptive trimming of the Hill estimator and detection of outliers in the extremes of heavy-tailed data

TL;DR

This paper proposes a robust, data-adaptive trimming method for the Hill estimator to accurately estimate the tail index of heavy-tailed distributions and detect outliers in extreme data, improving robustness and efficiency.

Contribution

It introduces a novel, adaptive trimming approach for the Hill estimator that is robust to outliers and contamination, with proven asymptotic properties and practical outlier detection capabilities.

Findings

Estimator is asymptotically normal under second order regular variation.

Method is minimax rate-optimal in the Hall class of distributions.

Successfully identifies outliers in real heavy-tailed data sets.

Abstract

We introduce a trimmed version of the Hill estimator for the index of a heavy-tailed distribution, which is robust to perturbations in the extreme order statistics. In the ideal Pareto setting, the estimator is essentially finite-sample efficient among all unbiased estimators with a given strict upper break-down point. For general heavy-tailed models, we establish the asymptotic normality of the estimator under second order regular variation conditions and also show it is minimax rate-optimal in the Hall class of distributions. We also develop an automatic, data-driven method for the choice of the trimming parameter which yields a new type of robust estimator that can adapt to the unknown level of contamination in the extremes. This adaptive robustness property makes our estimator particularly appealing and superior to other robust estimators in the setting where the extremes of the data are contaminated. As an important application of the data-driven selection of the trimming parameters, we obtain a methodology for the principled identification of extreme outliers in heavy tailed data. Indeed, the method has been shown to correctly identify the number of outliers in the previously explored Condroz data set.