On learning an interpreted language with recurrent models

TL;DR

This paper investigates the ability of recurrent neural networks like LSTM and GRU to learn language understanding by testing them on simplified datasets that mimic core properties of natural language, such as recursion and compositionality.

Contribution

It demonstrates that LSTM and GRU models can generalize to compositional interpretation under optimal training conditions, highlighting the importance of curriculum and data directionality.

Findings

LSTM and GRU generalize well with proper curriculum

Left-to-right composition improves learning

Extensive training data enhances generalization

Abstract

Can recurrent neural nets, inspired by human sequential data processing, learn to understand language? We construct simplified datasets reflecting core properties of natural language as modeled in formal syntax and semantics: recursive syntactic structure and compositionality. We find LSTM and GRU networks to generalise to compositional interpretation well, but only in the most favorable learning settings, with a well-paced curriculum, extensive training data, and left-to-right (but not right-to-left) composition.