The Nature of NLP: Analyzing Contributions in NLP Papers

TL;DR

This paper introduces a taxonomy and dataset for analyzing NLP research contributions, along with a model to automatically classify contribution types, revealing evolving research trends over decades.

Contribution

It presents a novel taxonomy, a large annotated dataset, and an automatic classification task for NLP paper contributions, enabling detailed analysis of research trends.

Findings

NLP research shifted focus from language-centric to dataset and method contributions.

The proposed model effectively classifies contribution types from abstracts.

Research trends show a resurgence of human-centric studies since the late 2010s.

Abstract

Natural Language Processing (NLP) is an established and dynamic field. Despite this, what constitutes NLP research remains debated. In this work, we address the question by quantitatively examining NLP research papers. We propose a taxonomy of research contributions and introduce NLPContributions, a dataset of nearly $2k$ NLP research paper abstracts, carefully annotated to identify scientific contributions and classify their types according to this taxonomy. We also introduce a novel task of automatically identifying contribution statements and classifying their types from research papers. We present experimental results for this task and apply our model to $\sim$$29k$ NLP research papers to analyze their contributions, aiding in the understanding of the nature of NLP research. We show that NLP research has taken a winding path -- with the focus on language and human-centric studies being prominent in the 1970s and 80s, tapering off in the 1990s and 2000s, and starting to rise again since the late 2010s. Alongside this revival, we observe a steady rise in dataset and methodological contributions since the 1990s, such that today, on average, individual NLP papers contribute in more ways than ever before. Our dataset and analyses offer a powerful lens for tracing research trends and offer potential for generating informed, data-driven literature surveys.