Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale

TL;DR

This paper advocates for using large language models to generate and process thick descriptions, enabling scalable representation of human cultural meaning in AI systems, moving beyond traditional numerical metrics.

Contribution

It introduces a framework for leveraging LLMs to automate thick descriptions, addressing limitations of standard metrics in capturing human cultural context at scale.

Findings

LLMs can generate verbal descriptions that retain cultural context.

Thick descriptions enable richer representation of human meaning.

Challenges include preserving context and interpretive pluralism.

Abstract

This position paper argues that large language models (LLMs) can make cultural context, and therefore human meaning, legible at an unprecedented scale in AI-based sociotechnical systems. We argue that such systems have previously been unable to represent human meaning because they rely on thin descriptions (numerical representations that enforce standardization and therefore strip human activity of the cultural context which gives it meaning). By contrast, scholars in the humanities and qualitative social sciences have developed frameworks for representing meaning through thick description (verbal representations that accommodate heterogeneity and retain contextual information needed to represent human meaning). The verbal capabilities of LLMs now provide a means of at least partially automating the generation and processing of thick descriptions, offering new ways to deploy them at scale. We argue that the problem of rendering human meaning legible is not just about selecting better metrics but about developing new representational formats based on thick description. We frame this as a crucial direction for the application of generative AI and identify five key challenges: preserving context, maintaining interpretive pluralism, integrating perspectives based on lived experience and critical distance, distinguishing qualitative content from quantitative magnitude, and acknowledging meaning as dynamic rather than static.