Learning to Compose Words into Sentences with Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning approach to automatically learn task-specific tree structures for sentence representation, outperforming traditional models and discovering some linguistically meaningful patterns.

Contribution

It presents a novel method for learning tree-structured neural networks optimized for downstream tasks without relying on explicit syntactic annotations.

Findings

Learned trees improve sentence encoding performance

Discovered linguistically meaningful but different structures from traditional syntax

Outperformed sequential and treebank-based models

Abstract

We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences. In contrast with prior work on tree-structured models in which the trees are either provided as input or predicted using supervision from explicit treebank annotations, the tree structures in this work are optimized to improve performance on a downstream task. Experiments demonstrate the benefit of learning task-specific composition orders, outperforming both sequential encoders and recursive encoders based on treebank annotations. We analyze the induced trees and show that while they discover some linguistically intuitive structures (e.g., noun phrases, simple verb phrases), they are different than conventional English syntactic structures.