Detecting AI-Generated Paraphrases in Bengali: A Comparative Study of Zero-Shot and Fine-Tuned Transformers

TL;DR

This study evaluates transformer models for detecting AI-generated Bengali text, showing that fine-tuning significantly enhances accuracy from near chance to over 90%, thus advancing Bengali AI text detection capabilities.

Contribution

It provides the first comprehensive comparison of zero-shot and fine-tuned transformer models for Bengali AI text detection, highlighting the importance of task-specific fine-tuning.

Findings

Zero-shot models perform near chance levels (~50% accuracy).

Fine-tuning improves performance to around 91% accuracy and F1-score.

IndicBERT shows limited effectiveness after fine-tuning.

Abstract

Large language models (LLMs) can produce text that closely resembles human writing. This capability raises concerns about misuse, including disinformation and content manipulation. Detecting AI-generated text is essential to maintain authenticity and prevent malicious applications. Existing research has addressed detection in multiple languages, but the Bengali language remains largely unexplored. Bengali's rich vocabulary and complex structure make distinguishing human-written and AI-generated text particularly challenging. This study investigates five transformer-based models: XLMRoBERTa-Large, mDeBERTaV3-Base, BanglaBERT-Base, IndicBERT-Base and MultilingualBERT-Base. Zero-shot evaluation shows that all models perform near chance levels (around 50% accuracy) and highlight the need for task-specific fine-tuning. Fine-tuning significantly improves performance, with XLM-RoBERTa, mDeBERTa and MultilingualBERT achieving around 91% on both accuracy and F1-score. IndicBERT demonstrates comparatively weaker performance, indicating limited effectiveness in fine-tuning for this task. This work advances AI-generated text detection in Bengali and establishes a foundation for building robust systems to counter AI-generated content.