Comparative Analysis of Large Language Models in Generating Telugu Responses for Maternal Health Queries

TL;DR

This study evaluates the performance of ChatGPT-4o, GeminiAI, and Perplexity AI in generating Telugu responses for maternal health queries, highlighting the importance of language and model choice in low-resource settings.

Contribution

It provides a comparative analysis of multiple LLMs in Telugu maternal health queries, emphasizing the need for improved regional language support in healthcare AI.

Findings

GeminiAI produces the most accurate and coherent Telugu responses.

Perplexity AI performs well with Telugu prompts.

ChatGPT's performance needs enhancement in this context.

Abstract

Large Language Models (LLMs) have been progressively exhibiting there capabilities in various areas of research. The performance of the LLMs in acute maternal healthcare area, predominantly in low resource languages like Telugu, Hindi, Tamil, Urdu etc are still unstudied. This study presents how ChatGPT-4o, GeminiAI, and Perplexity AI respond to pregnancy related questions asked in different languages. A bilingual dataset is used to obtain results by applying the semantic similarity metrics (BERT Score) and expert assessments from expertise gynecologists. Multiple parameters like accuracy, fluency, relevance, coherence and completeness are taken into consideration by the gynecologists to rate the responses generated by the LLMs. Gemini excels in other LLMs in terms of producing accurate and coherent pregnancy relevant responses in Telugu, while Perplexity demonstrated well when the prompts were in Telugu. ChatGPT's performance can be improved. The results states that both selecting an LLM and prompting language plays a crucial role in retrieving the information. Altogether, we emphasize for the improvement of LLMs assistance in regional languages for healthcare purposes.