Generalized Convergence Analysis of Tsetlin Machines: A Probabilistic Approach to Concept Learning

TL;DR

This paper provides a comprehensive probabilistic convergence analysis of Tsetlin Machines, establishing conditions under which they reliably learn conjunctions of literals, thereby enhancing their theoretical foundation and practical robustness.

Contribution

It introduces a novel Probabilistic Concept Learning framework and proves convergence for Tsetlin automaton-based algorithms in generalized settings with multiple features.

Findings

Proves convergence of Tsetlin automaton-based algorithms for conjunctions with feature count > 2

Introduces a simplified PCL framework with dedicated feedback and inclusion/exclusion probabilities

Establishes theoretical conditions (0.5<p<1) for reliable convergence

Abstract

Tsetlin Machines (TMs) have garnered increasing interest for their ability to learn concepts via propositional formulas and their proven efficiency across various application domains. Despite this, the convergence proof for the TMs, particularly for the AND operator (\emph{conjunction} of literals), in the generalized case (inputs greater than two bits) remains an open problem. This paper aims to fill this gap by presenting a comprehensive convergence analysis of Tsetlin automaton-based Machine Learning algorithms. We introduce a novel framework, referred to as Probabilistic Concept Learning (PCL), which simplifies the TM structure while incorporating dedicated feedback mechanisms and dedicated inclusion/exclusion probabilities for literals. Given $n$ features, PCL aims to learn a set of conjunction clauses $C_i$ each associated with a distinct inclusion probability $p_i$. Most importantly, we establish a theoretical proof confirming that, for any clause $C_k$, PCL converges to a conjunction of literals when $0.5<p_k<1$. This result serves as a stepping stone for future research on the convergence properties of Tsetlin automaton-based learning algorithms. Our findings not only contribute to the theoretical understanding of Tsetlin Machines but also have implications for their practical application, potentially leading to more robust and interpretable machine learning models.