Learning from Task Descriptions

TL;DR

This paper introduces a new framework and dataset, ZEST, for training NLP systems to solve unseen tasks by understanding descriptions, highlighting a large gap in current models' generalization abilities.

Contribution

The paper presents a novel framework and the ZEST dataset for evaluating NLP models' ability to learn new tasks from descriptions, emphasizing systematic generalization.

Findings

T5 model scores only 12% on ZEST

ZEST effectively tests systematic generalization

Framework enables training models on task descriptions

Abstract

Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.