Examining the development of attitude scales using Large Language Models (LLMs)

TL;DR

This study investigates using Large Language Models to evaluate attitude scale items, comparing AI assessments with human judges, and finds AI can match human judgment on most items, suggesting a new approach for scale development.

Contribution

It introduces a novel method of integrating AI LLMs into psychometric scale development, demonstrating comparable performance to human judges.

Findings

AI matched human judgments on 35 items

Minor differences observed in 23 items

Major differences found in 5 items

Abstract

For nearly a century, social researchers and psychologists have debated the efficacy of psychometric scales for attitude measurement, focusing on Thurstone's equal appearing interval scales and Likert's summated rating scales. Thurstone scales fell out of favour due to the labour intensive process of gathering judges' opinions on the initial items. However, advancements in technology have mitigated these challenges, nullifying the simplicity advantage of Likert scales, which have their own methodological issues. This study explores a methodological experiment to develop a Thurstone scale for assessing attitudes towards individuals living with AIDS. An electronic questionnaire was distributed to a group of judges, including undergraduate, postgraduate, and PhD students from disciplines such as social policy, law, medicine, and computer engineering, alongside established social researchers, and their responses were statistically analysed. The primary innovation of this study is the incorporation of an Artificial Intelligence (AI) Large Language Model (LLM) to evaluate the initial 63 items, comparing its assessments with those of the human judges. Interestingly, the AI provided also detailed explanations for its categorisation. Results showed no significant difference between AI and human judges for 35 items, minor differences for 23 items, and major differences for 5 items. This experiment demonstrates the potential of integrating AI with traditional psychometric methods to enhance the development of attitude measurement scales.