Adapting the Hill estimator to distributed inference: dealing with the bias

TL;DR

This paper introduces a bias correction method for the distributed Hill estimator, improving the accuracy of extreme value index estimation in distributed data settings.

Contribution

The paper develops an asymptotically unbiased distributed Hill estimator with bias correction, tailored for distributed inference scenarios.

Findings

Bias correction reduces asymptotic bias in distributed Hill estimator.

The corrected estimator maintains advantages of bias correction in extreme value analysis.

Applicable to distributed data stored across multiple machines.

Abstract

The distributed Hill estimator is a divide-and-conquer algorithm for estimating the extreme value index when data are stored in multiple machines. In applications, estimates based on the distributed Hill estimator can be sensitive to the choice of the number of the exceedance ratios used in each machine. Even when choosing the number at a low level, a high asymptotic bias may arise. We overcome this potential drawback by designing a bias correction procedure for the distributed Hill estimator, which adheres to the setup of distributed inference. The asymptotically unbiased distributed estimator we obtained, on the one hand, is applicable to distributed stored data, on the other hand, inherits all known advantages of bias correction methods in extreme value statistics.