Automata Extraction from Transformers

TL;DR

This paper introduces a novel automata extraction method for Transformer models, enabling interpretation of their processing of formal languages and improving transparency in understanding their operational mechanisms.

Contribution

It presents a new algorithm that treats Transformers as black boxes and extracts deterministic finite automata to interpret their language processing capabilities.

Findings

Transformers can be interpreted as automata using the proposed method.

The approach reveals how Transformers understand formal language structures.

The method enhances interpretability of Transformer models in language tasks.

Abstract

In modern machine (ML) learning systems, Transformer-based architectures have achieved milestone success across a broad spectrum of tasks, yet understanding their operational mechanisms remains an open problem. To improve the transparency of ML systems, automata extraction methods, which interpret stateful ML models as automata typically through formal languages, have proven effective for explaining the mechanism of recurrent neural networks (RNNs). However, few works have been applied to this paradigm to Transformer models. In particular, understanding their processing of formal languages and identifying their limitations in this area remains unexplored. In this paper, we propose an automata extraction algorithm specifically designed for Transformer models. Treating the Transformer model as a black-box system, we track the model through the transformation process of their internal latent representations during their operations, and then use classical pedagogical approaches like L* algorithm to interpret them as deterministic finite-state automata (DFA). Overall, our study reveals how the Transformer model comprehends the structure of formal languages, which not only enhances the interpretability of the Transformer-based ML systems but also marks a crucial step toward a deeper understanding of how ML systems process formal languages. Code and data are available at https://github.com/Zhang-Yihao/Transfomer2DFA.