How to quantify direct correlations between variables

TL;DR

This paper reviews and proposes new measures for quantifying direct correlations between variables, distinguishing them from indirect correlations, with applications to real datasets and a decision-making model.

Contribution

It introduces Jensen-Shannon-based regularized measures for direct correlation, addressing limitations of Kullback-Leibler-based measures and analyzing their properties.

Findings

Proposed measures are bounded in [0,1], avoiding KL divergence singularities.

Upper bounds of measures depend on alphabet size, aiding interpretation.

Illustrated measures on real datasets and a toy model with confidence intervals.

Abstract

Analyzing correlation between variables is often both the tool and the goal of modern science. A crucial question is whether the correlation between two variables is a direct correlation or only an indirect correlation through a confounder. We review the existing measures of direct correlation and organize them into two families, each corresponding to a systematic construction: (i) removing the direct correlation from the original joint distribution and quantifying the resulting distributional shift, and (ii) intervening on one variable via do-calculus and quantifying how the distribution of the other variable responds. For every Kullback--Leibler-based measure in either family, we propose a Jensen--Shannon-based regularized analogue. Since the square root of the Jensen--Shannon divergence is a bounded metric, the regularized measures take values in $[0,1]$ and are free of the singularity of the Kullback--Leibler divergence. We further analyze the achievable upper bound of each regularized measure under the observed marginal $p(x,z)$, which depends on the alphabet size and is in general strictly below $1$; this sets the correct scale against which observed values should be read. The properties and the differences of the proposed measures are illustrated on a decision-making toy model and on three public real datasets: Titanic survival, UCI Adult (Census Income), and the UC~Berkeley 1973 graduate admissions. Bootstrap $95\%$ confidence intervals are reported for every numerical value.