How Ready Are Generative Pre-trained Large Language Models for Explaining Bengali Grammatical Errors?

TL;DR

This paper evaluates the effectiveness of various large language models in explaining Bengali grammatical errors, highlighting current limitations and advocating for human oversight to enhance educational feedback in low-resource language settings.

Contribution

Introduces a Bengali GEE dataset and benchmarks multiple LLMs, revealing their limitations and emphasizing the need for human intervention to improve grammatical error explanations.

Findings

Current LLMs have limited accuracy in Bengali GEE.

Human oversight significantly improves explanation quality.

Benchmark dataset enables future research in Bengali GEC.

Abstract

Grammatical error correction (GEC) tools, powered by advanced generative artificial intelligence (AI), competently correct linguistic inaccuracies in user input. However, they often fall short in providing essential natural language explanations, which are crucial for learning languages and gaining a deeper understanding of the grammatical rules. There is limited exploration of these tools in low-resource languages such as Bengali. In such languages, grammatical error explanation (GEE) systems should not only correct sentences but also provide explanations for errors. This comprehensive approach can help language learners in their quest for proficiency. Our work introduces a real-world, multi-domain dataset sourced from Bengali speakers of varying proficiency levels and linguistic complexities. This dataset serves as an evaluation benchmark for GEE systems, allowing them to use context information to generate meaningful explanations and high-quality corrections. Various generative pre-trained large language models (LLMs), including GPT-4 Turbo, GPT-3.5 Turbo, Text-davinci-003, Text-babbage-001, Text-curie-001, Text-ada-001, Llama-2-7b, Llama-2-13b, and Llama-2-70b, are assessed against human experts for performance comparison. Our research underscores the limitations in the automatic deployment of current state-of-the-art generative pre-trained LLMs for Bengali GEE. Advocating for human intervention, our findings propose incorporating manual checks to address grammatical errors and improve feedback quality. This approach presents a more suitable strategy to refine the GEC tools in Bengali, emphasizing the educational aspect of language learning.