Don't Let Your Likert Scales Grow Up To Be Visual Analog Scales: Understanding the Relationship Between Number of Response Categories and Measurement Error

TL;DR

This paper investigates the optimal number of response categories for scales like Likert and VAS, showing that measurement error influences whether more options improve reliability or cause degradation.

Contribution

It provides a simulation-based analysis of how measurement error affects the optimal number of response categories in rating scales.

Findings

Reliability increases with response options if measurement error is constant.

When measurement error increases with categories, an optimal number exists.

Converting Likert to VAS without re-validation reduces reliability.

Abstract

The use of Visual Analog Scales (VAS), which can be broadly conceptualized as items where the response scale is 0-100, has surged recently due to the convenience of digital assessments. However, there is no consensus as to whether the use of VAS scales is optimal in a measurement sense. Put differently, in the 90+ years since Likert introduced his eponymous scale, the field does not know how to determine the optimal number of response options for a given item. In the current work, we investigate the optimal number of response categories using a series of simulations. We find that when the measurement error of an item is not dependent on the number of response categories, there is no true optimum; rather, reliability increases with number of response options and then plateaus. However, under the more realistic assumption that the measurement error of an item increases with the number of response categories, we find a clear optimum that depends on the rate of that increase. If measurement error increases with the number of response categories, then conversion of any Likert scale item to VAS will result in a drastic decrease in reliability. Finally, if researchers do want to change the response scale of a validated measure, they must re-validate the new measure as the measurement error of the scale is likely to change.