Estimators of Entropy and Information via Inference in Probabilistic Models

TL;DR

This paper introduces EEVI, a new method for estimating entropy and information bounds in high-dimensional probabilistic models, using importance sampling techniques that adapt to the model for high accuracy.

Contribution

The paper proposes EEVI, a novel entropy estimator that provides bounds via inference and importance sampling, applicable to arbitrary variables in probabilistic models.

Findings

EEVI delivers tight bounds on information quantities in complex models.

Demonstrated scalability on medical diagnosis and metabolic modeling tasks.

Improves decision-making by identifying most informative tests and measurement times.

Abstract

Estimating information-theoretic quantities such as entropy and mutual information is central to many problems in statistics and machine learning, but challenging in high dimensions. This paper presents estimators of entropy via inference (EEVI), which deliver upper and lower bounds on many information quantities for arbitrary variables in a probabilistic generative model. These estimators use importance sampling with proposal distribution families that include amortized variational inference and sequential Monte Carlo, which can be tailored to the target model and used to squeeze true information values with high accuracy. We present several theoretical properties of EEVI and demonstrate scalability and efficacy on two problems from the medical domain: (i) in an expert system for diagnosing liver disorders, we rank medical tests according to how informative they are about latent diseases, given a pattern of observed symptoms and patient attributes; and (ii) in a differential equation model of carbohydrate metabolism, we find optimal times to take blood glucose measurements that maximize information about a diabetic patient's insulin sensitivity, given their meal and medication schedule.