An experimental approach: Converting verbal expressions to numerical scales

TL;DR

This study explores converting verbal expressions into numerical scales for decision-making, demonstrating that verbal differences are often underestimated by common scales and analyzing respondent consistency in color preference evaluations.

Contribution

The paper introduces an experimental method for converting verbal expressions into numerical scales, highlighting their differences from traditional scales and analyzing respondent consistency.

Findings

Verbal expressions have smaller differences than common numerical scales.

Most respondents show consistent answers upon repeated questioning.

Inconsistent respondents tend to have more variable answers.

Abstract

One of the key issues in decision problems is the selection and use of the appropriate response scale. In this paper verbal expressions are converted into numerical scales for a subjective problem instance. The main motivation for our research was that verbal values in decision tasks are often mechanically converted into numbers, which thus typically do not fully represent the respondent's true evaluation. In our experiment, we conducted a color selection test with 462 subjects by testing six colors (red, green, blue, magenta, turquoise, yellow) defined from the Color Namer database on color-calibrated tablets in ISO standardized sensory test booths of a sensory laboratory. The colors were evaluated both in a pairwise comparison matrix (indirect ranking with four-item verbal category scale) and on a direct scoring basis. We determined scales that provide the closest results on average and individually to the direct scoring, based on the eigenvector and the logarithmic least squares methods. All results show that the difference between verbal expressions is much smaller than the one used by most of the common numerical scales. The respondents' inconsistency was also analyzed, even with a repeated question regarding their preference between a given pair of colors. It is shown that most decision makers answer fairly similarly for the second time, but there can be significant (even ordinal) differences. The respondents whose answers are further from the original tend to be more inconsistent in general.