A Nonparametric Statistical Approach to Content Analysis of Items

TL;DR

This paper introduces a nonparametric statistical method to objectively evaluate the consistency of specialists' judgments in content analysis of items for psychometric assessments, enhancing reliability in multidimensional construct evaluation.

Contribution

It develops a novel nonparametric statistical test to assess whether specialists have consistent judgment capabilities in content analysis, reducing subjectivity.

Findings

The proposed test effectively detects inconsistencies among specialists.

Simulation results show high accuracy of the test in various scenarios.

Application to real data demonstrates practical utility.

Abstract

In order to use psychometric instruments to assess a multidimensional construct, we may decompose it in dimensions and, in order to assess each dimension, develop a set of items, so one may assess the construct as a whole, by assessing its dimensions. In this scenario, the content analysis of items aims to verify if the developed items are assessing the dimension they are supposed to. In the content analysis process, it is customary to request the judgement of specialists in the studied construct about the dimension that the developed items assess, what makes it a subjective process as it relies upon the personal opinion of the specialists. This paper aims to develop a nonparametric statistical approach to the content analysis of items in order to present a practical method to assess the consistency of the content analysis process, by the development of a statistical test that seeks to determine if all the specialists have the same capability to judge the items. A simulation study is conducted to assess the consistency of the test and it is applied to a real validation process.