Trust, Safety, and Accuracy: Assessing LLMs for Routine Maternity Advice

TL;DR

This study assesses the effectiveness of various large language models in providing reliable, understandable pregnancy-related health information in rural India, highlighting their potential to improve maternal health education where medical resources are scarce.

Contribution

It evaluates and compares multiple LLMs for maternal health advice, emphasizing their potential to enhance healthcare communication in underserved regions.

Findings

Perplexity AI closely matches expert semantics.

ChatGPT-4o produces clearer, more understandable responses.

LLMs could serve as scalable health education tools in rural areas.

Abstract

Access to reliable maternal healthcare information is a major challenge in rural India due to limited medical resources and infrastructure. With over 830 million internet users and nearly half of rural women online, digital tools offer new opportunities for health education. This study evaluates large language models (LLMs) like ChatGPT-4o, Perplexity AI, and GeminiAI to provide reliable and understandable pregnancy-related information. Seventeen pregnancy-focused questions were posed to each model and compared with responses from maternal health professionals. Evaluations used semantic similarity, noun overlap, and readability metrics to measure content quality. Results show Perplexity closely matched expert semantics, while ChatGPT-4o produced clearer, more understandable text with better medical terminology. As internet access grows in rural areas, LLMs could serve as scalable aids for maternal health education. The study highlights the need for AI tools that balance accuracy and clarity to improve healthcare communication in underserved regions.