An evaluation of Google Translate for Sanskrit to English translation via sentiment and semantic analysis

TL;DR

This paper evaluates Google Translate's Sanskrit to English translation quality using sentiment and semantic analysis of the Bhagavad Gita, revealing limitations in translating poetic and contextual language.

Contribution

It introduces a framework for assessing machine translation quality of Sanskrit using BERT-based models, comparing Google Translate with expert translations.

Findings

Low similarity in sentiment and semantic analysis between Google Translate and experts.

Google Translate struggles with poetic, metaphorical, and context-dependent Sanskrit words.

Framework can be applied to evaluate other languages in machine translation.

Abstract

Google Translate has been prominent for language translation; however, limited work has been done in evaluating the quality of translation when compared to human experts. Sanskrit one of the oldest written languages in the world. In 2022, the Sanskrit language was added to the Google Translate engine. Sanskrit is known as the mother of languages such as Hindi and an ancient source of the Indo-European group of languages. Sanskrit is the original language for sacred Hindu texts such as the Bhagavad Gita. In this study, we present a framework that evaluates the Google Translate for Sanskrit using the Bhagavad Gita. We first publish a translation of the Bhagavad Gita in Sanskrit using Google Translate. Our framework then compares Google Translate version of Bhagavad Gita with expert translations using sentiment and semantic analysis via BERT-based language models. Our results indicate that in terms of sentiment and semantic analysis, there is low level of similarity in selected verses of Google Translate when compared to expert translations. In the qualitative evaluation, we find that Google translate is unsuitable for translation of certain Sanskrit words and phrases due to its poetic nature, contextual significance, metaphor and imagery. The mistranslations are not surprising since the Bhagavad Gita is known as a difficult text not only to translate, but also to interpret since it relies on contextual, philosophical and historical information. Our framework lays the foundation for automatic evaluation of other languages by Google Translate