Discovering Potential Correlations via Hypercontractivity

TL;DR

This paper introduces a new measure called the hypercontractivity coefficient for discovering hidden or potential correlations between variables, which traditional measures often miss, and demonstrates its effectiveness through theoretical and empirical validation.

Contribution

The paper proposes the hypercontractivity coefficient as a novel measure for potential correlation, along with a new estimator and extensive experiments validating its advantages.

Findings

The hypercontractivity coefficient satisfies natural axioms for potential correlation.

The proposed estimator effectively discovers potential correlations in real datasets.

It outperforms existing correlation measures in statistical power for binary hypothesis testing.

Abstract

Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest. While existing correlation measures are suitable for discovering average correlation, they fail to discover hidden or potential correlations. To bridge this gap, (i) we postulate a set of natural axioms that we expect a measure of potential correlation to satisfy; (ii) we show that the rate of information bottleneck, i.e., the hypercontractivity coefficient, satisfies all the proposed axioms; (iii) we provide a novel estimator to estimate the hypercontractivity coefficient from samples; and (iv) we provide numerical experiments demonstrating that this proposed estimator discovers potential correlations among various indicators of WHO datasets, is robust in discovering gene interactions from gene expression time series data, and is statistically more powerful than the estimators for other correlation measures in binary hypothesis testing of canonical examples of potential correlations.