Correcting Low-Signal Sensitivity in the Deliberative Reason Index

TL;DR

This paper introduces a modified Deliberative Reason Index (DRI) that corrects for bias under low-signal conditions, improving its reliability in assessing coherence in deliberative settings.

Contribution

A new continuous penalty modification to the DRI is proposed, reducing bias and maintaining scale, especially in low-signal scenarios.

Findings

Monte Carlo simulations show bias increases with group size under low signal.

The modified DRI reduces false positives in low-signal conditions.

Empirical data confirms the modification preserves substantive inferences.

Abstract

The Deliberative Reason Index (DRI) is increasingly used to assess the coherence between considerations and preferences in deliberative settings, including applications to LLM-generated data. Under low-signal conditions, however, the standard DRI can produce inflated scores by treating near-zero correlations as evidence of consistency. Monte Carlo simulations across common study designs show that this bias increases with group size and yields positive values even under random response. A modified DRI is introduced that applies a continuous penalty to low-signal correlation pairs. The modification preserves the original scale and reduces exactly to the standard DRI when substantive signal is present. A threshold sensitivity analysis identifies {\tau}=0.2as the optimal parameter. An empirical check with archival deliberative data shows that substantive inferences remain unchanged. The modification improves the reliability and comparability of the DRI in low-signal settings.