PaLM: A Hybrid Parser and Language Model

TL;DR

PaLM is a hybrid neural language model with an attention-based parser component that improves language understanding and can be trained with or without syntactic annotations, outperforming strong baselines.

Contribution

This paper introduces PaLM, combining a neural language model with an attention-based parser, enabling unsupervised and supervised syntactic parsing within language modeling.

Findings

PaLM outperforms strong baseline models in language modeling tasks.

The attention weights in PaLM can be used to derive an unsupervised constituency parser.

Supervised training of the attention component with syntactic annotations further enhances performance.

Abstract

We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.