Human-Centred Evaluation of Text-to-Image Generation Models for Self-expression of Mental Distress: A Dataset Based on GPT-4o

TL;DR

This study evaluates AI-generated images' effectiveness in aiding self-expression of mental distress among international students, introducing a new dataset with human evaluations to support mental health communication research.

Contribution

It presents the first publicly available dataset of text-to-image generation in mental health, including human judgment scores, and analyzes prompt design impact on helpfulness.

Findings

Prompt design significantly influences perceived helpfulness.

The illustrator persona received the highest helpfulness ratings.

The dataset enables future research in mental health communication and AI evaluation.

Abstract

Effective communication is central to achieving positive healthcare outcomes in mental health contexts, yet international students often face linguistic and cultural barriers that hinder their communication of mental distress. In this study, we evaluate the effectiveness of AI-generated images in supporting self-expression of mental distress. To achieve this, twenty Chinese international students studying at UK universities were invited to describe their personal experiences of mental distress. These descriptions were elaborated using GPT-4o with four persona-based prompt templates rooted in contemporary counselling practice to generate corresponding images. Participants then evaluated the helpfulness of generated images in facilitating the expression of their feelings based on their original descriptions. The resulting dataset comprises 100 textual descriptions of mental distress, 400 generated images, and corresponding human evaluation scores. Findings indicate that prompt design substantially affects perceived helpfulness, with the illustrator persona achieving the highest ratings. This work introduces the first publicly available text-to-image evaluation dataset with human judgment scores in the mental health domain, offering valuable resources for image evaluation, reinforcement learning with human feedback, and multi-modal research on mental health communication.