Estimating Extreme Value Index by Subsampling for Massive Datasets with Heavy-Tailed Distributions

TL;DR

This paper introduces a subsampling-based estimator for the extreme value index in massive heavy-tailed datasets, providing a scalable, consistent, and asymptotically normal approach with demonstrated effectiveness through simulations and real data analysis.

Contribution

It proposes a novel subsampling method for estimating the extreme value index in large datasets, improving scalability and accuracy over traditional methods.

Findings

Estimator is consistent and asymptotically normal.

Method accurately estimates high quantiles and tail probabilities.

Simulation and real data confirm promising performance.

Abstract

Modern statistical analyses often encounter datasets with massive sizes and heavy-tailed distributions. For datasets with massive sizes, traditional estimation methods can hardly be used to estimate the extreme value index directly. To address the issue, we propose here a subsampling-based method. Specifically, multiple subsamples are drawn from the whole dataset by using the technique of simple random subsampling with replacement. Based on each subsample, an approximate maximum likelihood estimator can be computed. The resulting estimators are then averaged to form a more accurate one. Under appropriate regularity conditions, we show theoretically that the proposed estimator is consistent and asymptotically normal. With the help of the estimated extreme value index, we can estimate high-level quantiles and tail probabilities of a heavy-tailed random variable consistently. Extensive simulation experiments are provided to demonstrate the promising performance of our method. A real data analysis is also presented for illustration purpose.