Interpretive Efficiency: Information-Geometric Foundations of Data Usefulness

TL;DR

This paper introduces Interpretive Efficiency, a new information-theoretic metric grounded in information geometry, to quantify how effectively data representations support task-specific interpretability in machine learning.

Contribution

It formalizes a task-aware interpretive efficiency measure with theoretical foundations, estimation guarantees, and practical validation on image and signal tasks.

Findings

Recovers theoretical orderings of representations

Exposes redundancy masked by accuracy

Correlates with robustness

Abstract

Interpretability is central to trustworthy machine learning, yet existing metrics rarely quantify how effectively data support an interpretive representation. We propose Interpretive Efficiency, a normalized, task-aware functional that measures the fraction of task-relevant information transmitted through an interpretive channel. The definition is grounded in five axioms ensuring boundedness, Blackwell-style monotonicity, data-processing stability, admissible invariance, and asymptotic consistency. We relate the functional to mutual information and derive a local Fisher-geometric expansion, then establish asymptotic and finite-sample estimation guarantees using standard empirical-process tools. Experiments on controlled image and signal tasks demonstrate that the measure recovers theoretical orderings, exposes representational redundancy masked by accuracy, and correlates with robustness, making it a practical, theory-backed diagnostic for representation design.