A Simple Recurrent Unit with Reduced Tensor Product Representations

TL;DR

The paper introduces TPRU, a simple recurrent unit that explicitly incorporates structured symbolic representations using reduced Tensor Product Representations, enhancing interpretability and performance in natural language tasks.

Contribution

It proposes the TPRU, a novel recurrent unit that explicitly performs structural-role binding and unbinding, improving structured representation learning.

Findings

TPRU outperforms traditional units on multiple datasets.

Gradient analysis supports the model design.

Demonstrates interpretability through linguistically grounded studies.

Abstract

idely used recurrent units, including Long-short Term Memory (LSTM) and the Gated Recurrent Unit (GRU), perform well on natural language tasks, but their ability to learn structured representations is still questionable. Exploiting reduced Tensor Product Representations (TPRs) --- distributed representations of symbolic structure in which vector-embedded symbols are bound to vector-embedded structural positions --- we propose the TPRU, a simple recurrent unit that, at each time step, explicitly executes structural-role binding and unbinding operations to incorporate structural information into learning. A gradient analysis of our proposed TPRU is conducted to support our model design, and its performance on multiple datasets shows the effectiveness of our design choices. Furthermore, observations on a linguistically grounded study demonstrate the interpretability of our TPRU.