Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Play

TL;DR

This paper introduces a novel framework using risk scales and role-play to evaluate ethical decision-making and biases in large language models, aiming to enhance their safety and fairness.

Contribution

It applies cognitive science risk assessment tools to LLMs and develops a new Ethical Decision-Making Risk Attitude Scale (EDRAS) for in-depth bias detection.

Findings

LLMs exhibit distinct risk personalities across domains

Identified systematic biases towards different groups

Provided tools for bias mitigation and safety enhancement

Abstract

As Large Language Models (LLMs) become more prevalent, concerns about their safety, ethics, and potential biases have risen. Systematically evaluating LLMs' risk decision-making tendencies and attitudes, particularly in the ethical domain, has become crucial. This study innovatively applies the Domain-Specific Risk-Taking (DOSPERT) scale from cognitive science to LLMs and proposes a novel Ethical Decision-Making Risk Attitude Scale (EDRAS) to assess LLMs' ethical risk attitudes in depth. We further propose a novel approach integrating risk scales and role-playing to quantitatively evaluate systematic biases in LLMs. Through systematic evaluation and analysis of multiple mainstream LLMs, we assessed the "risk personalities" of LLMs across multiple domains, with a particular focus on the ethical domain, and revealed and quantified LLMs' systematic biases towards different groups. This research helps understand LLMs' risk decision-making and ensure their safe and reliable application. Our approach provides a tool for identifying and mitigating biases, contributing to fairer and more trustworthy AI systems. The code and data are available.