Assesment of Generative AI abilities to diagnose and propose treatment in comparison with psychiatrists from Poland and Tunisa

TL;DR

This study compares the diagnostic and treatment proposal abilities of generative AI with psychiatrists from Poland and Tunisia.

Contribution

The study introduces a novel comparison between generative AI and human psychiatrists using a Turing test framework.

Findings

Generative AI provided valid diagnoses, especially with newer versions of ChatGPT.

Treatment proposals from AI were less accurate compared to diagnoses.

Human psychiatrists were generally favored in assessments over AI.

Abstract

Increasing popularity of Generative AI systems such as GPT provides us with new dilemmas concerning the future of diagnosis and novel tools to improve daily psychiatrits’s work.

The aim of the study was to assess the abilities of generative AI to diagnose and propose treatment in comparison with real psychiatrists and performing a Turing test.

We examined the ability to diagnose and propose treatment of various Generative AI versions (CHatGPT/CHATGPTpro etc.) and then compare the results with 10 clinicians performing the same task. Then a group of 10 psychiatry specialists not involved in the first evaluation assessed wether the diagnose and treatment were established by Generative AI or a clinician.

The reults showed that the generative AI systems were able to provide valid diagnosis in most of the cases with favour to newer and most proficient version of CHATGPT. Proposed treatment results were less accurate. The comparison between human and AI group was hard to accurately acces, with tendency to favouring psychiatrists group assesment as the right decision.

There is huge need to further explore the possibilites and limitations of Generative AI use in psychiatry.

There is huge need to further explore the possibilites and limitations of Generative AI use in psychiatry.

None Declared