ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models

TL;DR

ValueBench is a comprehensive psychometric benchmark designed to evaluate the value orientations and understanding of large language models, aiding responsible AI deployment.

Contribution

It introduces the first extensive benchmark based on psychometric inventories to assess LLMs' value orientations and understanding in realistic human-AI interactions.

Findings

LLMs exhibit shared and distinctive value orientations.

LLMs can approximate expert conclusions in value-related tasks.

ValueBench is publicly accessible for further research.

Abstract

Large Language Models (LLMs) are transforming diverse fields and gaining increasing influence as human proxies. This development underscores the urgent need for evaluating value orientations and understanding of LLMs to ensure their responsible integration into public-facing applications. This work introduces ValueBench, the first comprehensive psychometric benchmark for evaluating value orientations and value understanding in LLMs. ValueBench collects data from 44 established psychometric inventories, encompassing 453 multifaceted value dimensions. We propose an evaluation pipeline grounded in realistic human-AI interactions to probe value orientations, along with novel tasks for evaluating value understanding in an open-ended value space. With extensive experiments conducted on six representative LLMs, we unveil their shared and distinctive value orientations and exhibit their ability to approximate expert conclusions in value-related extraction and generation tasks. ValueBench is openly accessible at https://github.com/Value4AI/ValueBench.