People's Perceptions Toward Bias and Related Concepts in Large Language Models: A Systematic Review

TL;DR

This systematic review analyzes empirical studies on how people perceive biases, stereotypes, and safety issues in large language models, highlighting diverse perceptions and influencing factors.

Contribution

It synthesizes empirical insights from 15 studies on user perceptions of LLM biases, social norms, and safety concerns, providing a comprehensive overview.

Findings

Perceptions vary across biases and application areas.

Users recognize both advantages and biases in LLMs.

Factors influencing perceptions include context and individual differences.

Abstract

Large language models (LLMs) have brought breakthroughs in tasks including translation, summarization, information retrieval, and language generation, gaining growing interest in the CHI community. Meanwhile, the literature shows researchers' controversial perceptions about the efficacy, ethics, and intellectual abilities of LLMs. However, we do not know how people perceive LLMs that are pervasive in everyday tools, specifically regarding their experience with LLMs around bias, stereotypes, social norms, or safety. In this study, we conducted a systematic review to understand what empirical insights papers have gathered about people's perceptions toward LLMs. From a total of 231 retrieved papers, we full-text reviewed 15 papers that recruited human evaluators to assess their experiences with LLMs. We report different biases and related concepts investigated by these studies, four broader LLM application areas, the evaluators' perceptions toward LLMs' performances including advantages, biases, and conflicting perceptions, factors influencing these perceptions, and concerns about LLM applications.