Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses

TL;DR

This paper introduces a scalable, interpretable two-phase training method for word embeddings using the Tsetlin Machine, improving scalability and maintaining interpretability while achieving competitive results in NLP tasks.

Contribution

The paper presents a novel two-phase training approach for Tsetlin Machine-based word embeddings, addressing scalability issues and enhancing interpretability in NLP applications.

Findings

The method achieves competitive performance on benchmark datasets.

It maintains interpretability of the embeddings.

The approach is effective for sentiment analysis on IMDB.

Abstract

The Tsetlin Machine (TM) architecture has recently demonstrated effectiveness in Machine Learning (ML), particularly within Natural Language Processing (NLP). It has been utilized to construct word embedding using conjunctive propositional clauses, thereby significantly enhancing our understanding and interpretation of machine-derived decisions. The previous approach performed the word embedding over a sequence of input words to consolidate the information into a cohesive and unified representation. However, that approach encounters scalability challenges as the input size increases. In this study, we introduce a novel approach incorporating two-phase training to discover contextual embeddings of input sequences. Specifically, this method encapsulates the knowledge for each input word within the dataset's vocabulary, subsequently constructing embeddings for a sequence of input words utilizing the extracted knowledge. This technique not only facilitates the design of a scalable model but also preserves interpretability. Our experimental findings revealed that the proposed method yields competitive performance compared to the previous approaches, demonstrating promising results in contrast to human-generated benchmarks. Furthermore, we applied the proposed approach to sentiment analysis on the IMDB dataset, where the TM embedding and the TM classifier, along with other interpretable classifiers, offered a transparent end-to-end solution with competitive performance.