Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

TL;DR

This paper summarizes a workshop and shared task focused on evaluating and improving the linguistic generalizability of NLP systems beyond their training data distributions.

Contribution

It introduces a shared task designed to test NLP systems' robustness and generalizability across diverse linguistic phenomena and discusses key insights and lessons learned.

Findings

Systems showed varying robustness to linguistic challenges

Shared task highlighted gaps in current NLP generalization capabilities

Lessons inform future development of more linguistically adaptable NLP models

Abstract

This paper presents a summary of the first Workshop on Building Linguistically Generalizable Natural Language Processing Systems, and the associated Build It Break It, The Language Edition shared task. The goal of this workshop was to bring together researchers in NLP and linguistics with a shared task aimed at testing the generalizability of NLP systems beyond the distributions of their training data. We describe the motivation, setup, and participation of the shared task, provide discussion of some highlighted results, and discuss lessons learned.