Understanding heavy tails in a bounded world or, is a truncated heavy tail heavy or not?

TL;DR

This paper investigates how truncating heavy-tailed distributions affects their properties, distinguishing between soft and hard truncation regimes, and provides methods for estimation and testing using real network data.

Contribution

It introduces a framework for understanding truncated heavy tails, defines regimes, and develops consistent estimation and testing procedures for tail behavior.

Findings

In soft truncation, heavy tail characteristics are preserved.

Hard truncation significantly reduces heavy tail features.

Methods successfully applied to network data sets.

Abstract

We address the important question of the extent to which random variables and vectors with truncated power tails retain the characteristic features of random variables and vectors with power tails. We define two truncation regimes, soft truncation regime and hard truncation regime, and show that, in the soft truncation regime, truncated power tails behave, in important respects, as if no truncation took place. On the other hand, in the hard truncation regime much of "heavy tailedness" is lost. We show how to estimate consistently the tail exponent when the tails are truncated, and suggest statistical tests to decide on whether the truncation is soft or hard. Finally, we apply our methods to two recent data sets arising from computer networks.