Are We Aligned? A Preliminary Investigation of the Alignment of Responsible AI Values between LLMs and Human Judgment

TL;DR

This study examines how well large language models' responsible AI values align with human judgments, revealing closer alignment with AI practitioners but notable gaps between claimed values and actual requirement prioritization.

Contribution

It provides a systematic evaluation of LLMs' value alignment with humans across multiple tasks, highlighting discrepancies and practical risks in software engineering applications.

Findings

LLMs align more with AI practitioners than the general public.

Inconsistencies exist between LLMs' claimed values and their requirement prioritization.

Highlighting the need for benchmarking and monitoring of AI value alignment.

Abstract

Large Language Models (LLMs) are increasingly employed in software engineering tasks such as requirements elicitation, design, and evaluation, raising critical questions regarding their alignment with human judgments on responsible AI values. This study investigates how closely LLMs' value preferences align with those of two human groups: a US-representative sample and AI practitioners. We evaluate 23 LLMs across four tasks: (T1) selecting key responsible AI values, (T2) rating their importance in specific contexts, (T3) resolving trade-offs between competing values, and (T4) prioritizing software requirements that embody those values. The results show that LLMs generally align more closely with AI practitioners than with the US-representative sample, emphasizing fairness, privacy, transparency, safety, and accountability. However, inconsistencies appear between the values that LLMs claim to uphold (Tasks 1-3) and the way they prioritize requirements (Task 4), revealing gaps in faithfulness between stated and applied behavior. These findings highlight the practical risk of relying on LLMs in requirements engineering without human oversight and motivate the need for systematic approaches to benchmark, interpret, and monitor value alignment in AI-assisted software development.