Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values
Jing Yao, Xiaoyuan Yi, Shitong Duan, Jindong Wang, Yuzhuo Bai, Muhua Huang, Peng Zhang, Tun Lu, Zhicheng Dou, Maosong Sun, Xing Xie

TL;DR
This paper introduces the Value Compass Benchmarks, a comprehensive platform for evaluating LLMs' values by clarifying underlying motivations, adapting to evolving models, and accounting for cultural value differences.
Contribution
It presents a novel evaluation framework with modules for value clarification, adaptive testing, and pluralistic value measurement, addressing limitations of existing benchmarks.
Findings
Provides a holistic view of LLMs' underlying values.
Develops an adaptive evaluation framework for evolving LLMs.
Introduces a metric for measuring value alignment considering cultural pluralism.
Abstract
As Large Language Models (LLMs) achieve remarkable breakthroughs, aligning their values with humans has become imperative for their responsible development and customized applications. However, there still lack evaluations of LLMs values that fulfill three desirable goals. (1) Value Clarification: We expect to clarify the underlying values of LLMs precisely and comprehensively, while current evaluations focus narrowly on safety risks such as bias and toxicity. (2) Evaluation Validity: Existing static, open-source benchmarks are prone to data contamination and quickly become obsolete as LLMs evolve. Additionally, these discriminative evaluations uncover LLMs' knowledge about values, rather than valid assessments of LLMs' behavioral conformity to values. (3) Value Pluralism: The pluralistic nature of human values across individuals and cultures is largely ignored in measuring LLMs value…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInnovation Policy and R&D · Research, Science, and Academia · Research Data Management Practices
MethodsFocus
