Investigating Context Effects in Similarity Judgements in Large Language Models

TL;DR

This paper investigates how large language models exhibit order bias in similarity judgments, replicating human studies to understand their alignment with human cognitive biases and implications for AI application design.

Contribution

It replicates a human study on order effects in similarity judgments with various LLMs, revealing their human-like biases and informing better AI application development.

Findings

LLMs exhibit order bias similar to humans in similarity judgments

Order effects vary across different LLM architectures and settings

Implications for designing AI systems aligned with human decision-making

Abstract

Large Language Models (LLMs) have revolutionised the capability of AI models in comprehending and generating natural language text. They are increasingly being used to empower and deploy agents in real-world scenarios, which make decisions and take actions based on their understanding of the context. Therefore researchers, policy makers and enterprises alike are working towards ensuring that the decisions made by these agents align with human values and user expectations. That being said, human values and decisions are not always straightforward to measure and are subject to different cognitive biases. There is a vast section of literature in Behavioural Science which studies biases in human judgements. In this work we report an ongoing investigation on alignment of LLMs with human judgements affected by order bias. Specifically, we focus on a famous human study which showed evidence of order effects in similarity judgements, and replicate it with various popular LLMs. We report the different settings where LLMs exhibit human-like order effect bias and discuss the implications of these findings to inform the design and development of LLM based applications.