Objectifying the Subjective: Cognitive Biases in Topic Interpretations

TL;DR

This paper investigates how users interpret topics, revealing cognitive biases like anchoring heuristics, and proposes a new theory for understanding and evaluating topic interpretation grounded in cognitive psychology.

Contribution

It introduces a theory of topic interpretation based on heuristics and cognitive biases, supported by user studies and thematic analysis, to improve evaluation measures.

Findings

Users rely on availability and representativeness heuristics.

Interpretation involves anchoring on salient words and semantic adjustments.

Cognitive biases influence how topics are understood and evaluated.

Abstract

Interpretation of topics is crucial for their downstream applications. State-of-the-art evaluation measures of topic quality such as coherence and word intrusion do not measure how much a topic facilitates the exploration of a corpus. To design evaluation measures grounded on a task, and a population of users, we do user studies to understand how users interpret topics. We propose constructs of topic quality and ask users to assess them in the context of a topic and provide rationale behind evaluations. We use reflexive thematic analysis to identify themes of topic interpretations from rationales. Users interpret topics based on availability and representativeness heuristics rather than probability. We propose a theory of topic interpretation based on the anchoring-and-adjustment heuristic: users anchor on salient words and make semantic adjustments to arrive at an interpretation. Topic interpretation can be viewed as making a judgment under uncertainty by an ecologically rational user, and hence cognitive biases aware user models and evaluation frameworks are needed.