FarExStance: Explainable Stance Detection for Farsi

TL;DR

This paper presents FarExStance, a new dataset for explainable stance detection in Farsi, and evaluates various models including fine-tuned RoBERTa and large language models on their ability to detect stance and generate explanations.

Contribution

Introduction of FarExStance, a novel dataset for Farsi explainable stance detection, and comprehensive evaluation of multiple models including fine-tuned and large language models.

Findings

Fine-tuned RoBERTa achieved the highest stance detection accuracy.

Few-shot Claude-3.5-Sonnet provided the best explanations according to human evaluation.

GPT-4o generated the most coherent explanations in automatic metrics.

Abstract

We introduce FarExStance, a new dataset for explainable stance detection in Farsi. Each instance in this dataset contains a claim, the stance of an article or social media post towards that claim, and an extractive explanation which provides evidence for the stance label. We compare the performance of a fine-tuned multilingual RoBERTa model to several large language models in zero-shot, few-shot, and parameter-efficient fine-tuned settings on our new dataset. On stance detection, the most accurate models are the fine-tuned RoBERTa model, the LLM Aya-23-8B which has been fine-tuned using parameter-efficient fine-tuning, and few-shot Claude-3.5-Sonnet. Regarding the quality of the explanations, our automatic evaluation metrics indicate that few-shot GPT-4o generates the most coherent explanations, while our human evaluation reveals that the best Overall Explanation Score (OES) belongs to few-shot Claude-3.5-Sonnet. The fine-tuned Aya-32-8B model produced explanations most closely aligned with the reference explanations.