Towards Synthesizing Normative Data for Cognitive Assessments Using Generative Multimodal Large Language Models

TL;DR

This paper explores using advanced generative multimodal large language models to synthesize normative data for cognitive assessments, aiming to overcome traditional data collection challenges and facilitate the development of new image-based tests.

Contribution

It demonstrates the feasibility of using GPT-4o models with refined prompting strategies to generate synthetic normative responses for cognitive tests, capturing demographic and diagnostic variations.

Findings

Advanced prompts improve differentiation between diagnostic groups.

BERTScore is the most reliable metric for evaluating responses.

Synthetic responses exhibit higher realism and diversity.

Abstract

Cognitive assessments require normative data as essential benchmarks for evaluating individual performance. Hence, developing new cognitive tests based on novel image stimuli is challenging due to the lack of readily available normative data. Traditional data collection methods are costly, time-consuming, and infrequently updated, limiting their practical utility. Recent advancements in generative multimodal large language models (MLLMs) offer a new approach to generate synthetic normative data from existing cognitive test images. We investigated the feasibility of using MLLMs, specifically GPT-4o and GPT-4o-mini, to synthesize normative textual responses for established image-based cognitive assessments, such as the "Cookie Theft" picture description task. Two distinct prompting strategies-naive prompts with basic instructions and advanced prompts enriched with contextual guidance-were evaluated. Responses were analyzed using embeddings to assess their capacity to distinguish diagnostic groups and demographic variations. Performance metrics included BLEU, ROUGE, BERTScore, and an LLM-as-a-judge evaluation. Advanced prompting strategies produced synthetic responses that more effectively distinguished between diagnostic groups and captured demographic diversity compared to naive prompts. Superior models generated responses exhibiting higher realism and diversity. BERTScore emerged as the most reliable metric for contextual similarity assessment, while BLEU was less effective for evaluating creative outputs. The LLM-as-a-judge approach provided promising preliminary validation results. Our study demonstrates that generative multimodal LLMs, guided by refined prompting methods, can feasibly generate robust synthetic normative data for existing cognitive tests, thereby laying the groundwork for developing novel image-based cognitive assessments without the traditional limitations.