Discovering the critical number of respondents to validate an item in a questionnaire: The Binomial Cut-level Content Validity proposal

TL;DR

This paper introduces a new statistical method to determine the minimum number of respondents needed to validate questionnaire items, addressing paradoxes and improving accuracy over existing content validity approaches.

Contribution

It proposes the Binomial Cut-level Content Validity method, which enhances item validation accuracy by addressing paradoxes and refining respondent sample size determination.

Findings

Identifies the minimum respondent number for item validation

Detects paradoxical validation situations

Provides more precise item retention recommendations

Abstract

The question that drives this research is: "How to discover the number of respondents that are necessary to validate items of a questionnaire as actually essential to reach the questionnaire's proposal?" Among the efforts in this subject, \cite{Lawshe1975, Wilson2012, Ayre_CVR_2014} approached this issue by proposing and refining the Content Validation Ratio (CVR) that looks to identify items that are actually essentials. Despite their contribution, these studies do not check if an item validated as "essential" should be also validated as "not essential" by the same sample, which should be a paradox. Another issue is the assignment a probability equal a 50\% to a item be randomly checked by a respondent as essential, despite an evaluator has three options to choose. Our proposal faces these issues, making it possible to verify if a paradoxical situation occurs, and being more precise in recommending whether an item should either be retained or discarded from a questionnaire.