Cross-Language Assessment of Mathematical Capability of ChatGPT

TL;DR

This study evaluates ChatGPT's mathematical problem-solving abilities across multiple Indian languages, examining the impact of chain-of-thought prompting and identifying current limitations in regional language performance.

Contribution

It provides the first comprehensive assessment of ChatGPT's mathematical capabilities in regional Indian languages and analyzes the effect of chain-of-thought prompting in these contexts.

Findings

ChatGPT performs variably across languages, with lower accuracy in regional Indian languages.

Chain-of-thought prompting improves mathematical response accuracy in multiple languages.

The study highlights significant limitations in ChatGPT's regional language mathematical reasoning.

Abstract

This paper presents an evaluation of the mathematical capability of ChatGPT across diverse languages like Hindi, Gujarati, and Marathi. ChatGPT, based on GPT-3.5 by OpenAI, has garnered significant attention for its natural language understanding and generation abilities. However, its performance in solving mathematical problems across multiple natural languages remains a comparatively unexplored area, especially in regional Indian languages. In this paper, we explore those capabilities as well as using chain-of-thought prompting to figure out if it increases the accuracy of responses as much as it does in the English language and provide insights into the current limitations.