Keeping Users Engaged During Repeated Administration of the Same Questionnaire: Using Large Language Models to Reliably Diversify Questions

TL;DR

This study demonstrates that large language models can generate diverse questionnaire variants that maintain validity, reduce respondent fatigue, and improve engagement during repeated administrations in longitudinal research.

Contribution

The paper introduces a method using LLMs to create reliable, diverse questionnaire variants that preserve psychometric properties, enhancing engagement in repeated assessments.

Findings

LLM-generated variants showed consistent validity with standard questionnaires.

Participants found LLM variants less repetitive and more engaging.

Psychometric tests confirmed the reliability of LLM-generated question variants.

Abstract

Standardized, validated questionnaires are vital tools in research and healthcare, offering dependable self-report data. Prior work has revealed that virtual agent-administered questionnaires are almost equivalent to self-administered ones in an electronic form. Despite being an engaging method, repeated use of virtual agent-administered questionnaires in longitudinal or pre-post studies can induce respondent fatigue, impacting data quality via response biases and decreased response rates. We propose using large language models (LLMs) to generate diverse questionnaire versions while retaining good psychometric properties. In a longitudinal study, participants interacted with our agent system and responded daily for two weeks to one of the following questionnaires: a standardized depression questionnaire, question variants generated by LLMs, or question variants accompanied by LLM-generated small talk. The responses were compared to a validated depression questionnaire. Psychometric testing revealed consistent covariation between the external criterion and focal measure administered across the three conditions, demonstrating the reliability and validity of the LLM-generated variants. Participants found that the variants were significantly less repetitive than repeated administrations of the same standardized questionnaire. Our findings highlight the potential of LLM-generated variants to invigorate agent-administered questionnaires and foster engagement and interest, without compromising their validity.