# ChatGPT and Gemini in warfarin counseling

## TL;DR

This study compares ChatGPT and Gemini in answering patient questions about warfarin, finding both reliable but with differences in detail and clarity.

## Contribution

The novel contribution is a direct comparison of two LLMs for warfarin counseling, evaluating scientific adequacy, clarity, and accuracy.

## Key findings

- ChatGPT responses were shorter but scientifically more adequate compared to Gemini.
- Gemini scored higher in clarity and user-friendly communication.
- Both models provided reliable information but required expert supervision for safe guidance.

## Abstract

To compare the accuracy, scientific adequacy, and clarity of responses provided by ChatGPT-4o and Gemini to frequently asked patients’ questions about warfarin use.

Forty patients’ questions were posed to ChatGPT and Gemini using the zero-shot method. Four cardiovascular surgeons evaluated the responses for adequacy, scientific accuracy, and clarity on a 5-point Likert scale. The first and second set of data evaluations were separated by seven days to minimize any memory effect. The experts were blinded to the type of LLM that created the response.

ChatGPT responses were significantly shorter (187.3 ± 47.6 vs 291.4 ± 98.1 words; P < 0.001) but scientifically more adequate (4.38 ± 0.30 vs 4.17 ± 0.35; P = 0.004). Gemini scored higher in terms of clarity (4.69 ± 0.24 vs 4.48 ± 0.33; P < 0.001). The two models did not significantly differ in terms of accuracy (P = 0.606).

Both LLMs provide reliable and understandable information for warfarin counseling. While ChatGPT provides dense and scientifically detailed responses, Gemini excels in clarity and user-friendly communication. However, expert supervision and appropriate clinical guidance are critical for safe and comprehensive guidance.