Evaluation of Google Translate for Mandarin Chinese translation using sentiment and semantic analysis

TL;DR

This paper assesses Google Translate's Mandarin Chinese translation quality using sentiment and semantic analysis, revealing limitations in translating cultural and historical nuances compared to human experts.

Contribution

It introduces an automated framework combining sentiment and semantic analysis to evaluate machine translation quality against human translations for Mandarin Chinese.

Findings

Google Translate shows variable semantic and sentiment accuracy.

It struggles with translating Chinese cultural allusions.

Mistranslations are linked to lack of contextual and historical understanding.

Abstract

Machine translation using large language models (LLMs) is having a significant global impact, making communication easier. Mandarin Chinese is the official language used for communication by the government and media in China. In this study, we provide an automated assessment of translation quality of Google Translate with human experts using sentiment and semantic analysis. In order to demonstrate our framework, we select the classic early twentieth-century novel 'The True Story of Ah Q' with selected Mandarin Chinese to English translations. We use Google Translate to translate the given text into English and then conduct a chapter-wise sentiment analysis and semantic analysis to compare the extracted sentiments across the different translations. Our results indicate that the precision of Google Translate differs both in terms of semantic and sentiment analysis when compared to human expert translations. We find that Google Translate is unable to translate some of the specific words or phrases in Chinese, such as Chinese traditional allusions. The mistranslations may be due to lack of contextual significance and historical knowledge of China.