A Comparative Study of Feature Selection in Tsetlin Machines

TL;DR

This paper evaluates various feature selection methods for Tsetlin Machines, demonstrating that TM-internal scorers are effective and computationally efficient, and establishing a baseline for interpretability in TM models.

Contribution

It adapts and benchmarks multiple feature selection techniques for Tsetlin Machines, including novel TM-specific scorers, and assesses their effectiveness across diverse datasets.

Findings

TM-internal scorers perform competitively in feature importance estimation.

Simpler TM-specific scorers retain accuracy with lower computational cost.

The study provides the first comprehensive baseline for feature selection in Tsetlin Machines.

Abstract

Feature Selection (FS) is crucial for improving model interpretability, reducing complexity, and sometimes for enhancing accuracy. The recently introduced Tsetlin machine (TM) offers interpretable clause-based learning, but lacks established tools for estimating feature importance. In this paper, we adapt and evaluate a range of FS techniques for TMs, including classical filter and embedded methods as well as post-hoc explanation methods originally developed for neural networks (e.g., SHAP and LIME) and a novel family of embedded scorers derived from TM clause weights and Tsetlin automaton (TA) states. We benchmark all methods across 12 datasets, using evaluation protocols, like Remove and Retrain (ROAR) strategy and Remove and Debias (ROAD), to assess causal impact. Our results show that TM-internal scorers not only perform competitively but also exploit the interpretability of clauses to reveal interacting feature patterns. Simpler TM-specific scorers achieve similar accuracy retention at a fraction of the computational cost. This study establishes the first comprehensive baseline for FS in TM and paves the way for developing specialized TM-specific interpretability techniques.