Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality

TL;DR

This paper introduces a novel method using recursive neural networks with bottlenecks to measure and analyze the compositionality of data in NLP tasks, applicable to both synthetic and natural language datasets.

Contribution

It proposes a new compositionality metric based on comparing model representations with and without bottlenecks, enabling ranking of data samples by their compositionality.

Findings

Bottleneck compression affects non-compositional data more significantly.

The BCM metric effectively distinguishes between compositional and non-compositional samples.

The approach provides a new tool for analyzing compositionality in NLP datasets.

Abstract

A recent line of work in NLP focuses on the (dis)ability of models to generalise compositionally for artificial languages. However, when considering natural language tasks, the data involved is not strictly, or locally, compositional. Quantifying the compositionality of data is a challenging task, which has been investigated primarily for short utterances. We use recursive neural models (Tree-LSTMs) with bottlenecks that limit the transfer of information between nodes. We illustrate that comparing data's representations in models with and without the bottleneck can be used to produce a compositionality metric. The procedure is applied to the evaluation of arithmetic expressions using synthetic data, and sentiment classification using natural language data. We demonstrate that compression through a bottleneck impacts non-compositional examples disproportionately and then use the bottleneck compositionality metric (BCM) to distinguish compositional from non-compositional samples, yielding a compositionality ranking over a dataset.