Omni TM-AE: A Scalable and Interpretable Embedding Model Using the Full Tsetlin Machine State Space

TL;DR

Omni TM-AE is a scalable, interpretable embedding model leveraging the Tsetlin Machine's full state space, achieving competitive performance in NLP tasks while maintaining transparency and reusability.

Contribution

It introduces a novel embedding approach that fully utilizes the Tsetlin Machine's state matrix, enhancing interpretability and scalability in NLP embeddings.

Findings

Performs competitively with mainstream models in semantic tasks.

Enables reusable, interpretable embeddings in a single training phase.

Balances performance, scalability, and interpretability without opaque architectures.

Abstract

The increasing complexity of large-scale language models has amplified concerns regarding their interpretability and reusability. While traditional embedding models like Word2Vec and GloVe offer scalability, they lack transparency and often behave as black boxes. Conversely, interpretable models such as the Tsetlin Machine (TM) have shown promise in constructing explainable learning systems, though they previously faced limitations in scalability and reusability. In this paper, we introduce Omni Tsetlin Machine AutoEncoder (Omni TM-AE), a novel embedding model that fully exploits the information contained in the TM's state matrix, including literals previously excluded from clause formation. This method enables the construction of reusable, interpretable embeddings through a single training phase. Extensive experiments across semantic similarity, sentiment classification, and document clustering tasks show that Omni TM-AE performs competitively with and often surpasses mainstream embedding models. These results demonstrate that it is possible to balance performance, scalability, and interpretability in modern Natural Language Processing (NLP) systems without resorting to opaque architectures.