Towards the Next-generation Bayesian Network Classifiers

TL;DR

This paper introduces NeuralKDB, a neural network-based high-order Bayesian classifier that learns distributional feature representations to better model complex dependencies, significantly improving classification performance on diverse datasets.

Contribution

The paper proposes a novel neural extension of the K-dependence Bayesian classifier that learns feature value representations to capture high-order dependencies, overcoming traditional Bayesian network limitations.

Findings

NeuralKDB outperforms traditional Bayesian classifiers on 60 UCI datasets.

The model effectively captures high-order feature dependencies.

NeuralKDB surpasses other neural classifiers without distributional representations.

Abstract

Bayesian network classifiers provide a feasible solution to tabular data classification, with a number of merits like high time and memory efficiency, and great explainability. However, due to the parameter explosion and data sparsity issues, Bayesian network classifiers are restricted to low-order feature dependency modeling, making them struggle in extrapolating the occurrence probabilities of complex real-world data. In this paper, we propose a novel paradigm to design high-order Bayesian network classifiers, by learning distributional representations for feature values, as what has been done in word embedding and graph representation learning. The learned distributional representations are encoded with the semantic relatedness between different features through their observed co-occurrence patterns in training data, which then serve as a hallmark to extrapolate the occurrence probabilities of new test samples. As a classifier design realization, we remake the K-dependence Bayesian classifier (KDB) by extending it into a neural version, i.e., NeuralKDB, where a novel neural network architecture is designed to learn distributional representations of feature values and parameterize the conditional probabilities between interdependent features. A stochastic gradient descent based algorithm is designed to train the NeuralKDB model efficiently. Extensive classification experiments on 60 UCI datasets demonstrate that the proposed NeuralKDB classifier excels in capturing high-order feature dependencies and significantly outperforms the conventional Bayesian network classifiers, as well as other competitive classifiers, including two neural network based classifiers without distributional representation learning.