# ChatGPT as a mental health advisory service: Comparing evaluations from youth and health professionals

## TL;DR

This study compares how young people and health professionals evaluate mental health advice from ChatGPT and professionals, finding youth prefer ChatGPT's answers for their clarity and usefulness.

## Contribution

The first blind evaluation comparing youth and professional perspectives on ChatGPT's mental health advice, proposing a hybrid advisory model.

## Key findings

- Young people rated ChatGPT's answers higher for relevance and utility compared to health professionals' answers.
- Health professionals showed no strong preference and were more critical of ChatGPT's answers, citing lack of empathy.
- A hybrid model combining LLMs and professional expertise is proposed to improve mental health advisory services.

## Abstract

Despite the growing use of artificial intelligence in youth mental health support, little is known about how either young people or health professionals perceive answers to mental health-related inquiries generated by large language models (LLMs). Therefore, we draw on Media Richness Theory to examine how these two user groups perceive the “richness” of text-based communication in this context and whether young people and health professionals differ in their assessment.

A total of 123 young people and 31 health professionals evaluated answers to youth mental health inquiries. Each inquiry had two blinded answers: one generated by ChatGPT (GPT-4) and one written by a health professional. Participants rated the answers for validation, relevance, clarity, and utility and were asked to recommend one or both answers. Open-ended responses elaborating participant choices were also collected.

The quantitative findings show that young people and health professionals rated answers from both sources similarly on validation, clarity, and utility. However, young people rated ChatGPT's answers higher for relevance and utility, finding them “richer.” This was supported by qualitative data, where youth preferred ChatGPT's clear and actionable answers. Health professionals showed no strong preference and were more critical, often finding the answers too detailed or lacking empathy.

This study is the first to compare youth and professional perspectives on ChatGPT's role in youth mental health advice within a blind evaluation design. We conclude by proposing a hybrid advisory model that combines professional expertise with LLMs to enhance the efficiency, scale, and accessibility of youth mental health advisory services.