Embedding Symbolic Temporal Knowledge into Deep Sequential Models

TL;DR

This paper introduces a method to embed symbolic temporal knowledge into deep neural networks using automata and graph neural networks, enhancing performance in robot sequence tasks especially with limited data.

Contribution

It proposes a novel approach to incorporate linear temporal logic into deep models via automata embeddings, bridging structured knowledge and data-driven learning.

Findings

Embeddings improve sequence recognition accuracy.

Method enhances imitation learning performance.

Automata-based embeddings facilitate knowledge transfer.

Abstract

Sequences and time-series often arise in robot tasks, e.g., in activity recognition and imitation learning. In recent years, deep neural networks (DNNs) have emerged as an effective data-driven methodology for processing sequences given sufficient training data and compute resources. However, when data is limited, simpler models such as logic/rule-based methods work surprisingly well, especially when relevant prior knowledge is applied in their construction. However, unlike DNNs, these "structured" models can be difficult to extend, and do not work well with raw unstructured data. In this work, we seek to learn flexible DNNs, yet leverage prior temporal knowledge when available. Our approach is to embed symbolic knowledge expressed as linear temporal logic (LTL) and use these embeddings to guide the training of deep models. Specifically, we construct semantic-based embeddings of automata generated from LTL formula via a Graph Neural Network. Experiments show that these learnt embeddings can lead to improvements in downstream robot tasks such as sequential action recognition and imitation learning.