Leveraging Order-Theoretic Tournament Graphs for Assessing Internal Consistency in Survey-Based Instruments Across Diverse Scenarios

TL;DR

This paper presents Monotone Delta, an order-theoretic measure that improves the assessment of internal consistency in survey instruments by addressing limitations of traditional reliability metrics, especially in complex, real-world scenarios.

Contribution

It introduces Monotone Delta, a novel, assumption-free measure based on order theory, to evaluate survey reliability more accurately across diverse and challenging conditions.

Findings

Monotone Delta outperforms traditional metrics in stability across scenarios

It effectively handles multidimensional and non-normal data distributions

The method is applicable in psychology, healthcare, and human factors contexts

Abstract

This paper introduces Monotone Delta, an order-theoretic measure designed to enhance the reliability assessment of survey-based instruments in human-machine interactions. Traditional reliability measures, such as Cronbach's Alpha and McDonald's Omega, often yield misleading estimates due to their sensitivity to redundancy, multidimensional constructs, and assumptions of normality and uncorrelated errors. These limitations can compromise decision-making in human-centric evaluations, where survey instruments inform adaptive interfaces, cognitive workload assessments, and human-AI trust models. Monotone Delta addresses these issues by quantifying internal consistency through the minimization of ordinal contradictions and alignment with a unidimensional latent order using weighted tournaments. Unlike traditional approaches, it operates without parametric or model-based assumptions. We conducted theoretical analyses and experimental evaluations on four challenging scenarios: tau-equivalence, redundancy, multidimensionality, and non-normal distributions, and proved that Monotone Delta provides more stable reliability assessments compared to existing methods. The Monotone Delta is a valuable alternative for evaluating questionnaire-based assessments in psychology, human factors, healthcare, and interactive system design, enabling organizations to optimize survey instruments, reduce costly redundancies, and enhance confidence in human-system interactions.