Information Theoretic Structure Learning with Confidence

TL;DR

This paper introduces a new nonparametric structure discovery method using weighted ensemble divergence estimators that achieve parametric convergence rates and enable statistical validation.

Contribution

It proposes a novel approach for nonparametric structure learning that overcomes slow convergence issues in continuous models by employing weighted ensemble divergence estimators.

Findings

Achieves parametric convergence rates in nonparametric settings.

Obeys an asymptotic central limit theorem for hypothesis testing.

Facilitates statistical validation of learned structures.

Abstract

Information theoretic measures (e.g. the Kullback Liebler divergence and Shannon mutual information) have been used for exploring possibly nonlinear multivariate dependencies in high dimension. If these dependencies are assumed to follow a Markov factor graph model, this exploration process is called structure discovery. For discrete-valued samples, estimates of the information divergence over the parametric class of multinomial models lead to structure discovery methods whose mean squared error achieves parametric convergence rates as the sample size grows. However, a naive application of this method to continuous nonparametric multivariate models converges much more slowly. In this paper we introduce a new method for nonparametric structure discovery that uses weighted ensemble divergence estimators that achieve parametric convergence rates and obey an asymptotic central limit theorem that facilitates hypothesis testing and other types of statistical validation.