Counting Distinctions: On the Conceptual Foundations of Shannon's Information Theory

TL;DR

This paper explores the conceptual foundations of Shannon's information theory through the lens of logical entropy, which counts distinctions in partitions, offering a dual perspective to traditional subset-based probability.

Contribution

It introduces a logical entropy framework based on distinctions in partitions, providing a conceptual underpinning for Shannon's information theory.

Findings

Logical entropy counts normalized distinctions in partitions.

Logical theory offers a dual perspective to Shannon's entropy.

Provides a conceptual foundation linking logic and information theory.

Abstract

Categorical logic has shown that modern logic is essentially the logic of subsets (or "subobjects"). Partitions are dual to subsets so there is a dual logic of partitions where a "distinction" [an ordered pair of distinct elements (u,u') from the universe U ] is dual to an "element". An element being in a subset is analogous to a partition p on U making a distinction, i.e., if u and u' were in different blocks of p. Subset logic leads to finite probability theory by taking the (Laplacian) probability as the normalized size of each subset-event of a finite universe. The analogous step in the logic of partitions is to assign to a partition the number of distinctions made by a partition normalized by the total number of ordered pairs |UxU| from the finite universe. That yields a notion of "logical entropy" for partitions and a "logical information theory." The logical theory directly counts the (normalized) number of distinctions in a partition while Shannon's theory gives the average number of binary partitions needed to make those same distinctions. Thus the logical theory is seen as providing a conceptual underpinning for Shannon's theory based on the logical notion of "distinctions." (forthcoming in Synthese)