XChoice: Explainable Evaluation of AI-Human Alignment in LLM-based Constrained Choice Decision Making

TL;DR

XChoice is an explainable framework that evaluates AI-human alignment in decision making by fitting interpretable models to data, revealing nuanced differences and guiding improvements beyond simple accuracy metrics.

Contribution

The paper introduces XChoice, a novel mechanism-based evaluation method that captures decision factors and trade-offs, enabling detailed analysis of AI-human alignment.

Findings

Heterogeneous alignment across models and activities

Misalignment concentrated in Black and married groups

Robustness validated through invariance analysis

Abstract

We present XChoice, an explainable framework for evaluating AI-human alignment in constrained decision making. Moving beyond outcome agreement such as accuracy and F1 score, XChoice fits a mechanism-based decision model to human data and LLM-generated decisions, recovering interpretable parameters that capture the relative importance of decision factors, constraint sensitivity, and implied trade-offs. Alignment is assessed by comparing these parameter vectors across models, options, and subgroups. We demonstrate XChoice on Americans' daily time allocation using the American Time Use Survey (ATUS) as human ground truth, revealing heterogeneous alignment across models and activities and salient misalignment concentrated in Black and married groups. We further validate robustness of XChoice via an invariance analysis and evaluate targeted mitigation with a retrieval augmented generation (RAG) intervention. Overall, XChoice provides mechanism-based metrics that diagnose misalignment and support informed improvements beyond surface outcome matching.