On the Rigour of Scientific Writing: Criteria, Analysis, and Insights

TL;DR

This paper presents a data-driven, domain-agnostic framework for automatically identifying and assessing scientific rigour criteria in research papers, demonstrating its effectiveness across ML and NLP venues.

Contribution

It introduces a novel, automated approach to model and evaluate scientific rigour, addressing the lack of computational tools for this purpose.

Findings

Rigour framing certainty enhances perceived scientific rigour.

Suggestion certainty and probability uncertainty reduce perceived rigour.

Framework effectively models rigour across different scientific domains.

Abstract

Rigour is crucial for scientific research as it ensures the reproducibility and validity of results and findings. Despite its importance, little work exists on modelling rigour computationally, and there is a lack of analysis on whether these criteria can effectively signal or measure the rigour of scientific papers in practice. In this paper, we introduce a bottom-up, data-driven framework to automatically identify and define rigour criteria and assess their relevance in scientific writing. Our framework includes rigour keyword extraction, detailed rigour definition generation, and salient criteria identification. Furthermore, our framework is domain-agnostic and can be tailored to the evaluation of scientific rigour for different areas, accommodating the distinct salient criteria across fields. We conducted comprehensive experiments based on datasets collected from two high impact venues for Machine Learning and NLP (i.e., ICLR and ACL) to demonstrate the effectiveness of our framework in modelling rigour. In addition, we analyse linguistic patterns of rigour, revealing that framing certainty is crucial for enhancing the perception of scientific rigour, while suggestion certainty and probability uncertainty diminish it.