We Should Evaluate Real-World Impact

TL;DR

The paper highlights the scarcity of real-world impact evaluations in NLP research and argues for increased focus on assessing practical effects to enhance usefulness and adoption.

Contribution

It provides a structured survey showing the minimal presence of impact evaluations in NLP papers and advocates for more comprehensive real-world impact assessments.

Findings

Only about 0.1% of papers include impact evaluations

Most impact evaluations are superficial and metric-focused

Emphasizes the need for thorough real-world impact assessments

Abstract

The ACL community has very little interest in evaluating the real-world impact of NLP systems. A structured survey of the ACL Anthology shows that perhaps 0.1% of its papers contain such evaluations; furthermore most papers which include impact evaluations present them very sketchily and instead focus on metric evaluations. NLP technology would be more useful and more quickly adopted if we seriously tried to understand and evaluate its real-world impact.