Recursive Top-Down Production for Sentence Generation with Latent Trees

TL;DR

This paper introduces a recursive top-down production model for sentence generation using latent trees, employing a dynamic programming algorithm to marginalize over tree structures, and demonstrates its effectiveness on synthetic and translation tasks.

Contribution

It presents a novel dynamic programming approach to model latent tree structures in sentence generation, improving performance on synthetic tasks and providing insights into learned tree structures.

Findings

Outperforms previous models on the SCAN LENGTH split

Performs comparably to models with ground-truth trees on question formation

Shows promising results on German-English translation

Abstract

We model the recursive production property of context-free grammars for natural and synthetic languages. To this end, we present a dynamic programming algorithm that marginalises over latent binary tree structures with $N$ leaves, allowing us to compute the likelihood of a sequence of $N$ tokens under a latent tree model, which we maximise to train a recursive neural function. We demonstrate performance on two synthetic tasks: SCAN (Lake and Baroni, 2017), where it outperforms previous models on the LENGTH split, and English question formation (McCoy et al., 2020), where it performs comparably to decoders with the ground-truth tree structure. We also present experimental results on German-English translation on the Multi30k dataset (Elliott et al., 2016), and qualitatively analyse the induced tree structures our model learns for the SCAN tasks and the German-English translation task.