Knowledge extraction from the learning of sequences in a long short term memory (LSTM) architecture

TL;DR

This paper presents a method to extract and validate automata from LSTM networks trained on sequence validity detection, enabling interpretability of learned sequence patterns.

Contribution

It introduces a clustering-based approach to derive automata from LSTM hidden states, bridging neural network learning with formal automata representation.

Findings

Successfully applied to artificial grammars and real-world data

Automata accurately predict sequence validity

Provides interpretability of LSTM sequence learning

Abstract

We introduce a general method to extract knowledge from a recurrent neural network (Long Short Term Memory) that has learnt to detect if a given input sequence is valid or not, according to an unknown generative automaton. Based on the clustering of the hidden states, we explain how to build and validate an automaton that corresponds to the underlying (unknown) automaton, and allows to predict if a given sequence is valid or not. The method is illustrated on artificial grammars (Reber's grammar variations) as well as on a real use-case whose underlying grammar is unknown.