AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP

TL;DR

This paper evaluates reasoning-based large language models on Arabic NLP tasks, highlighting the effectiveness of few-shot prompting, DeepSeek architectures, and fine-tuning strategies in improving performance on complex linguistic tasks.

Contribution

It introduces DeepSeek models for Arabic NLP and systematically benchmarks their reasoning capabilities across multiple tasks with various prompting and fine-tuning methods.

Findings

Few-shot prompting significantly boosts classification accuracy.

DeepSeek models outperform baseline GPT models on inference tasks.

LoRA fine-tuning further enhances model performance.

Abstract

Large language models (LLMs) have shown remarkable progress in reasoning abilities and general natural language processing (NLP) tasks, yet their performance on Arabic data, characterized by rich morphology, diverse dialects, and complex script, remains underexplored. This paper presents a comprehensive benchmarking study of multiple reasoning-focused LLMs, with a special emphasis on the newly introduced DeepSeek models, across a suite of fifteen Arabic NLP tasks. We experiment with various strategies, including zero-shot, few-shot, and fine-tuning. This allows us to systematically evaluate performance on datasets covering a range of applications to examine their capacity for linguistic reasoning under different levels of complexity. Our experiments reveal several key findings. First, carefully selecting just three in-context examples delivers an average uplift of over 13 F1 points on classification tasks-boosting sentiment analysis from 35.3% to 87.5% and paraphrase detection from 56.1% to 87.0%. Second, reasoning-focused DeepSeek architectures outperform a strong GPT o4-mini baseline by an average of 12 F1 points on complex inference tasks in the zero-shot setting. Third, LoRA-based fine-tuning yields up to an additional 8 points in F1 and BLEU compared to equivalent increases in model scale. The code is available at https://github.com/gufranSabri/deepseek-evals