Shoulders of Giants: A Look at the Degree and Utility of Openness in NLP Research

TL;DR

This paper examines the openness in NLP research by analyzing artefact sharing patterns, revealing disparities across venues and languages, and highlighting that a significant portion of papers do not fully share their resources.

Contribution

It provides a quantitative analysis of openness in NLP research, uncovering patterns and disparities in artefact sharing across venues and languages.

Findings

Over 30% of papers do not publicly release artefacts.

Different NLP venues show varying reuse patterns.

Significant language-wise disparities in artefact availability.

Abstract

We analysed a sample of NLP research papers archived in ACL Anthology as an attempt to quantify the degree of openness and the benefit of such an open culture in the NLP community. We observe that papers published in different NLP venues show different patterns related to artefact reuse. We also note that more than 30% of the papers we analysed do not release their artefacts publicly, despite promising to do so. Further, we observe a wide language-wise disparity in publicly available NLP-related artefacts.