Research Borderlands: Analysing Writing Across Research Cultures

TL;DR

This paper introduces a human-centered framework for analyzing and measuring research-specific cultural norms in academic writing, revealing LLMs' lack of cultural competence and tendency to homogenize research styles.

Contribution

It develops a novel framework and computational metrics to identify and compare cultural norms in research writing, and assesses LLMs' cultural competence in this context.

Findings

LLMs tend to homogenize research writing styles.

Research cultures exhibit distinct structural, stylistic, rhetorical, and citational norms.

The framework effectively surfaces latent cultural norms in research papers.

Abstract

Improving cultural competence of language technologies is important. However most recent works rarely engage with the communities they study, and instead rely on synthetic setups and imperfect proxies of culture. In this work, we take a human-centered approach to discover and measure language-based cultural norms, and cultural competence of LLMs. We focus on a single kind of culture, research cultures, and a single task, adapting writing across research cultures. Through a set of interviews with interdisciplinary researchers, who are experts at moving between cultures, we create a framework of structural, stylistic, rhetorical, and citational norms that vary across research cultures. We operationalise these features with a suite of computational metrics and use them for (a) surfacing latent cultural norms in human-written research papers at scale; and (b) highlighting the lack of cultural competence of LLMs, and their tendency to homogenise writing. Overall, our work illustrates the efficacy of a human-centered approach to measuring cultural norms in human-written and LLM-generated texts.