ChatGPT and Vaccine Hesitancy: A Comparison of English, Spanish, and French Responses Using a Validated Scale

TL;DR

This study evaluates ChatGPT's responses on vaccine hesitancy across English, Spanish, and French using a validated scale, revealing language-based differences and generally lower hesitancy than humans.

Contribution

It provides a cross-lingual analysis of ChatGPT's vaccine-related responses using a standardized hesitancy scale, highlighting variations and consistency across languages and model parameters.

Findings

ChatGPT responses show less vaccine hesitancy than human responses.

English responses are more hesitant than Spanish and French.

Responses are consistent across model parameters but vary across scale factors.

Abstract

ChatGPT is a popular information system (over 1 billion visits in August 2023) that can generate natural language responses to user queries. It is important to study the quality and equity of its responses on health-related topics, such as vaccination, as they may influence public health decision-making. We use the Vaccine Hesitancy Scale (VHS) proposed by Shapiro et al.1 to measure the hesitancy of ChatGPT responses in English, Spanish, and French. We find that: (a) ChatGPT responses indicate less hesitancy than those reported for human respondents in past literature; (b) ChatGPT responses vary significantly across languages, with English responses being the most hesitant on average and Spanish being the least; (c) ChatGPT responses are largely consistent across different model parameters but show some variations across the scale factors (vaccine competency, risk). Results have implications for researchers interested in evaluating and improving the quality and equity of health-related web information.