# Quality and Usability of Prostate Cancer Information Generated by Artificial Intelligence Chatbots: A Comparative Analysis

## TL;DR

This study compares how well AI chatbots provide prostate cancer information, finding that while the information is clear, it lacks actionable guidance for patients.

## Contribution

The study evaluates and compares the quality and usability of prostate cancer information from multiple AI chatbots using standardized tools.

## Key findings

- Most chatbots provided clear explanations of prostate cancer, but overall information quality was moderate.
- No chatbot consistently offered high-quality, actionable guidance for patients.
- Understandability was high, but actionability scores were uniformly poor across all platforms.

## Abstract

Artificial intelligence chatbots such as ChatGPT are increasingly used by patients to seek information about prostate cancer. In this study, we evaluated several commonly available AI chatbots to assess the quality, clarity, and usefulness of the information they provide. We found that while most chatbots explained prostate cancer in a clear and easy-to-understand manner, the overall quality of information was moderate. Importantly, very little practical guidance was provided to help patients understand what steps to take after receiving information. Differences in information quality were observed across platforms, but no chatbot consistently demonstrated high-quality, actionable patient guidance. These findings suggest that AI chatbots may be useful for basic education about prostate cancer but should not replace discussions with healthcare professionals or trusted patient information resources.

Background: Artificial intelligence chatbots are increasingly used by patients to obtain health information, including for prostate cancer. While these platforms offer accessible and conversational responses, concerns remain regarding the quality, usability, and clinical relevance of AI-generated content. This study comparatively evaluated patient-directed prostate cancer information generated by commonly used AI chatbots. Methods: Standardised prostate cancer-related prompts were developed using Google Trends and authoritative healthcare resources. Identical queries were submitted to five publicly accessible AI chatbots: ChatGPT 5.2, Google Gemini, Claude AI, Microsoft Copilot, and Perplexity. Responses were independently assessed by two blinded reviewers using the DISCERN instrument for information quality and the Patient Education Materials Assessment Tool for printable materials (PEMAT-P) for understandability and actionability. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Readability was evaluated using the Flesch–Kincaid Reading Ease score. Descriptive statistics were used for comparative and pooled analyses. Results: Overall information quality was moderate, with a pooled median (interquartile range [IQR]) DISCERN score of 56.5 (53.0–61.0). Higher mean DISCERN scores were observed for ChatGPT 5.2 and Microsoft Copilot, whereas lower scores were observed for Claude and Perplexity. PEMAT-P understandability was consistently high across platforms, with a pooled median (IQR) score of 91.7% (83.3–91.7%). In contrast, PEMAT-P actionability was uniformly poor, with a pooled median (IQR) score of 0% (0–0%). Readability analysis demonstrated moderate complexity, with a pooled median (IQR) Flesch–Kincaid Reading Ease score of 50.4 (49.2–52.5) and a median word count of 666 (657–1022). Inter-rater reliability was good for PEMAT understandability (ICC 0.841) and moderate for DISCERN (ICC 0.712). Conclusions: AI chatbots provide highly understandable but only moderately high-quality patient-directed prostate cancer information, with a consistent lack of actionable guidance. Although variation in content quality was observed across platforms, significant limitations remain in evidence transparency and practical patient support. Future development should prioritise integration of evidence-based resources and actionable decision-support tools to enhance the role of AI chatbots in prostate cancer education.