Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts

TL;DR

This paper introduces an automated system to analyze NLP research framing, revealing trends like increased scientific focus and support for human fact-checkers over full automation.

Contribution

It presents a novel three-component system that infers research framing elements and applies it to real NLP domains, improving analysis accuracy.

Findings

Identified a rise in vague research goals.

Detected increased emphasis on scientific exploration.

Observed a shift towards supporting human fact-checkers.

Abstract

Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated fact-checking using an existing dataset, and hate speech detection for which we annotate a new dataset-achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in vague or underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human fact-checkers rather than pursuing full automation.