Evaluating Telugu Proficiency in Large Language Models_ A Comparative Analysis of ChatGPT and Gemini

TL;DR

This study compares the Telugu language proficiency of ChatGPT and Gemini LLMs through a set of 20 questions, assessing their grammatical, vocabulary, and reasoning skills to evaluate their suitability for Telugu language tasks.

Contribution

It provides a comparative analysis of Telugu capabilities in ChatGPT and Gemini, highlighting strengths and weaknesses in handling grammar, vocabulary, and reasoning in Telugu.

Findings

ChatGPT shows better grammatical understanding

Gemini has a broader Telugu vocabulary

Both models exhibit limitations in complex reasoning

Abstract

The growing prominence of large language models (LLMs) necessitates the exploration of their capabilities beyond English. This research investigates the Telugu language proficiency of ChatGPT and Gemini, two leading LLMs. Through a designed set of 20 questions encompassing greetings, grammar, vocabulary, common phrases, task completion, and situational reasoning, the study delves into their strengths and weaknesses in handling Telugu. The analysis aims to identify the LLM that demonstrates a deeper understanding of Telugu grammatical structures, possesses a broader vocabulary, and exhibits superior performance in tasks like writing and reasoning. By comparing their ability to comprehend and use everyday Telugu expressions, the research sheds light on their suitability for real-world language interaction. Furthermore, the evaluation of adaptability and reasoning capabilities provides insights into how each LLM leverages Telugu to respond to dynamic situations. This comparative analysis contributes to the ongoing discussion on multilingual capabilities in AI and paves the way for future research in developing LLMs that can seamlessly integrate with Telugu-speaking communities.