A Partial Order on Uncertainty and Information

TL;DR

This paper investigates the relationship between information and uncertainty, providing conditions under which increased information reduces uncertainty for certain types of random variables and information measures.

Contribution

It establishes necessary and sufficient conditions for the reduction of uncertainty with increased information for continuous and integer-valued variables, and analyzes Shannon information.

Findings

Uncertainty reduction depends on specific conditions for continuous variables.

Similar conditions are derived for Shannon information.

The paper clarifies when information increases or decreases uncertainty.

Abstract

Information and uncertainty are closely related and extensively studied concepts in a number of scientific disciplines such as communication theory, probability theory, and statistics. Increasing the information arguably reduces the uncertainty on a given random subject. Consider the uncertainty measure as the variance of a random variable. Given the information that its outcome is in an interval, the uncertainty is expected to reduce when the interval shrinks. This proposition is not generally true. In this paper, we provide a necessary and sufficient condition for this proposition when the random variable is absolutely continuous or integer valued. We also give a similar result on Shannon information.