Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Models

TL;DR

This paper introduces the Alignment Score, a metric for evaluating how well large language models' multi-step reasoning aligns with human preferences, correlating with accuracy and coherence.

Contribution

It presents a novel semantic-level metric for assessing structured reasoning alignment and demonstrates its effectiveness in correlating with model performance and reasoning quality.

Findings

Alignment Score tracks task accuracy across models and reasoning depths.

Misalignment at greater depths is mainly due to thematic shift and redundant reasoning.

Alignment Score correlates with accuracy, readability, and coherence.

Abstract

This paper primarily demonstrates a method to quantitatively assess the alignment between multi-step, structured reasoning in large language models and human preferences. We introduce the Alignment Score, a semantic-level metric that compares a model-produced chain of thought traces with a human-preferred reference by constructing semantic-entropy-based matrices over intermediate steps and measuring their divergence. Our analysis shows that Alignment Score tracks task accuracy across models and hop depths, and peaks at 2-hop reasoning. Empirical results further indicate that misalignment at greater reasoning depths is driven mainly by alignment errors such as thematic shift and redundant reasoning. Viewing chain sampling as drawing from a distribution over reasoning paths, we empirically demonstrate a strong and consistent correlation between Alignment Score and accuracy, readability, and coherence, supporting its use as a diagnostic signal. The code is available.