Concerns on Bias in Large Language Models when Creating Synthetic Personae

TL;DR

This paper discusses the ethical, technical, and bias-related challenges of using synthetic personae generated by large language models in HCI research, emphasizing the need for thorough testing and bias mitigation.

Contribution

It highlights the existence of bias in black-box LLMs and explores methods to manipulate and test these models for creating synthetic personae in HCI.

Findings

Bias exists within black-box LLMs used for synthetic personae

Manipulation methods can influence LLM outputs

Thorough testing is essential before deploying LLM-generated personae

Abstract

This position paper explores the benefits, drawbacks, and ethical considerations of incorporating synthetic personae in HCI research, particularly focusing on the customization challenges beyond the limitations of current Large Language Models (LLMs). These perspectives are derived from the initial results of a sub-study employing vignettes to showcase the existence of bias within black-box LLMs and explore methods for manipulating them. The study aims to establish a foundation for understanding the challenges associated with these models, emphasizing the necessity of thorough testing before utilizing them to create synthetic personae for HCI research.