Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research

TL;DR

This paper critically examines the concepts of language tasks and language games in NLP, proposing a clearer framework to understand progress towards modeling general language competence.

Contribution

It formalizes the notions of language task and language game, providing a philosophical and methodological foundation for evaluating NLP research progress.

Findings

Clarifies the distinction between language tasks and language games.

Proposes a framework to assess progress in modeling language competence.

Highlights the implicit assumptions in current NLP research methodologies.

Abstract

"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language competence.