Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing

TL;DR

This paper investigates how simulating different Big Five personality traits in large language models can reduce priming biases in relevance judgments, leading to more reliable information retrieval evaluations.

Contribution

It introduces personality prompting as a novel method to mitigate threshold priming effects in LLM-based relevance labeling, grounded in psychological theory.

Findings

Certain personality profiles, like High Openness, reduce priming susceptibility.

The effectiveness of personality profiles varies across models and tasks.

Personality prompting can improve the reliability of LLM relevance judgments.

Abstract

Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices.