Entropy and the Kullback-Leibler Divergence for Bayesian Networks: Computational Complexity and Efficient Implementation

TL;DR

This paper introduces efficient algorithms for computing Shannon's entropy and Kullback-Leibler divergence in Bayesian networks, significantly reducing computational complexity, especially for Gaussian BNs, and demonstrating their practical effectiveness.

Contribution

It presents novel, efficient algorithms for entropy and KL divergence calculation in Bayesian networks, leveraging their structure to improve computational performance.

Findings

Reduced KL divergence computation from cubic to quadratic for Gaussian BNs

Provided numerical examples demonstrating algorithm efficiency

Enhanced understanding of entropy and divergence calculations in BNs

Abstract

Bayesian networks (BNs) are a foundational model in machine learning and causal inference. Their graphical structure can handle high-dimensional problems, divide them into a sparse collection of smaller ones, underlies Judea Pearl's causality, and determines their explainability and interpretability. Despite their popularity, there are almost no resources in the literature on how to compute Shannon's entropy and the Kullback-Leibler (KL) divergence for BNs under their most common distributional assumptions. In this paper, we provide computationally efficient algorithms for both by leveraging BNs' graphical structure, and we illustrate them with a complete set of numerical examples. In the process, we show it is possible to reduce the computational complexity of KL from cubic to quadratic for Gaussian BNs.