The Probabilistic Tsetlin Machine: A Novel Approach to Uncertainty Quantification

TL;DR

The paper introduces the Probabilistic Tsetlin Machine (PTM), a novel extension of Tsetlin Machines that quantifies uncertainty in predictions by learning probability distributions over automaton states, enhancing interpretability and reliability.

Contribution

This work presents the PTM framework that incorporates probability learning into Tsetlin Machines, enabling effective uncertainty quantification and decision boundary delineation.

Findings

PTM effectively quantifies uncertainty in noisy datasets.

PTM achieves competitive calibration and entropy metrics on benchmark datasets.

PTM provides interpretable uncertainty estimates comparable to Bayesian methods.

Abstract

Tsetlin Machines (TMs) have emerged as a compelling alternative to conventional deep learning methods, offering notable advantages such as smaller memory footprint, faster inference, fault-tolerant properties, and interpretability. Although various adaptations of TMs have expanded their applicability across diverse domains, a fundamental gap remains in understanding how TMs quantify uncertainty in their predictions. In response, this paper introduces the Probabilistic Tsetlin Machine (PTM) framework, aimed at providing a robust, reliable, and interpretable approach for uncertainty quantification. Unlike the original TM, the PTM learns the probability of staying on each state of each Tsetlin Automaton (TA) across all clauses. These probabilities are updated using the feedback tables that are part of the TM framework: Type I and Type II feedback. During inference, TAs decide their actions by sampling states based on learned probability distributions, akin to Bayesian neural networks when generating weight values. In our experimental analysis, we first illustrate the spread of the probabilities across TA states for the noisy-XOR dataset. Then we evaluate the PTM alongside benchmark models using both simulated and real-world datasets. The experiments on the simulated dataset reveal the PTM's effectiveness in uncertainty quantification, particularly in delineating decision boundaries and identifying regions of high uncertainty. Moreover, when applied to multiclass classification tasks using the Iris dataset, the PTM demonstrates competitive performance in terms of predictive entropy and expected calibration error, showcasing its potential as a reliable tool for uncertainty estimation. Our findings underscore the importance of selecting appropriate models for accurate uncertainty quantification in predictive tasks, with the PTM offering a particularly interpretable and effective solution.