Analyzing machine-learned representations: A natural language case study

TL;DR

This paper investigates how neural network representations of language compare to human cognition, revealing both similarities in abstract rule learning and differences in systematicity and generalization, informing AI development and psychological understanding.

Contribution

It introduces diagnostic tests for neural language representations, analyzes their systematicity, and explores how training data influences their ability to learn and generalize rules.

Findings

Neural models show some capacity for abstract rule learning.

Representations exhibit heuristic strategies and lack full systematicity.

Training data and augmentations affect generalization and heuristic use.

Abstract

As modern deep networks become more complex, and get closer to human-like capabilities in certain domains, the question arises of how the representations and decision rules they learn compare to the ones in humans. In this work, we study representations of sentences in one such artificial system for natural language processing. We first present a diagnostic test dataset to examine the degree of abstract composable structure represented. Analyzing performance on these diagnostic tests indicates a lack of systematicity in the representations and decision rules, and reveals a set of heuristic strategies. We then investigate the effect of the training distribution on learning these heuristic strategies, and study changes in these representations with various augmentations to the training set. Our results reveal parallels to the analogous representations in people. We find that these systems can learn abstract rules and generalize them to new contexts under certain circumstances -- similar to human zero-shot reasoning. However, we also note some shortcomings in this generalization behavior -- similar to human judgment errors like belief bias. Studying these parallels suggests new ways to understand psychological phenomena in humans as well as informs best strategies for building artificial intelligence with human-like language understanding.