Effective sample size approximations as entropy measures

TL;DR

This paper explores alternative effective sample size metrics for importance sampling, linking them to entropy measures like Re9nyi and Tsallis, and demonstrates their application in variable selection through numerical simulations.

Contribution

It establishes theoretical connections between ESS metrics and entropy measures, extending their applicability and analyzing their performance in importance sampling.

Findings

All ESS functions in Huggins-Roy's family satisfy desirable properties.

ESS formulas effectively approximate theoretical ESS in simulations.

Application of ESS formulas improves variable selection performance.

Abstract

In this work, we analyze alternative effective sample size (ESS) metrics for importance sampling algorithms, and discuss a possible extended range of applications. We show the relationship between the ESS expressions used in the literature and two entropy families, the R\'enyi and Tsallis entropy. The R\'enyi entropy is connected to the Huggins-Roy's ESS family introduced in \cite{Huggins15}. We prove that that all the ESS functions included in the Huggins-Roy's family fulfill all the desirable theoretical conditions. We analyzed and remark the connections with several other fields, such as the Hill numbers introduced in ecology, the Gini inequality coefficient employed in economics, and the Gini impurity index used mainly in machine learning, to name a few. Finally, by numerical simulations, we study the performance of different ESS expressions contained in the previous ESS families in terms of approximation of the theoretical ESS definition, and show the application of ESS formulas in a variable selection problem.