Learned practical guidelines for evaluating Conditional Entropy and Mutual Information in discovering major factors of response-vs-covariate dynamics

TL;DR

This paper develops practical, data-driven guidelines for evaluating conditional entropy and mutual information to identify key factors in response-vs-covariate dynamics without relying on explicit models, using categorical data analysis.

Contribution

It introduces a novel framework for analyzing Re-Co dynamics with entropy measures, providing computational guidelines that do not require consistent estimations and address curse of dimensionality.

Findings

Practical guidelines for evaluating CE and mutual information based on the confirmable criterion.

Application of the guidelines on contingency tables reduces effects of curse of dimensionality.

Six detailed examples demonstrating the effectiveness of the proposed approach.

Abstract

We reformulate and reframe a series of increasingly complex parametric statistical topics into a framework of response-vs-covariate (Re-Co) dynamics that is described without any explicit functional structures. Then we resolve these topics' data analysis tasks by discovering major factors underlying such Re-Co dynamics by only making use of data's categorical nature. The major factor selection protocol at the heart of Categorical Exploratory Data Analysis (CEDA) paradigm is illustrated and carried out by employing Shannon's conditional entropy (CE) and mutual information ($I[Re; Co] $) as two key Information Theoretical measurements. Through the process of evaluating these two entropy-based measurements and resolving statistical tasks, we acquire several computational guidelines for carrying out the major factor selection protocol in a do-and-learn fashion. Specifically, practical guidelines are established for evaluating CE and $I[Re; Co] $ in accord with the criterion called [C1:confirmable]. Via [C1:confirmable] criterion, we make no attempts on acquiring consistent estimations of these theoretical information measurements. All evaluations are carried out on a contingency table platform, upon which the practical guidelines also provide ways of lessening effects of curse of dimensionality. We explicitly carry out six examples of Re-Co dynamics, within each of which, several widely extended scenarios are also explored and discussed.