Learning and Evaluating General Linguistic Intelligence

TL;DR

This paper defines and evaluates general linguistic intelligence in models, analyzing their ability to adapt quickly to new language tasks, and introduces a new metric for measuring learning speed and generalization.

Contribution

It provides an empirical framework for assessing linguistic general intelligence and proposes a novel online encoding metric to evaluate learning efficiency.

Findings

Models require many in-domain examples to adapt.

Models are prone to catastrophic forgetting.

Models tend to overfit to dataset quirks.

Abstract

We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.