BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization
Ahmed Rafid, Rumman Adib, Fariya Ahmed, Ajwad Abrar, Mohammed Saidul Islam

TL;DR
BanglaSummEval is a novel reference-free, question-answering-based framework that evaluates factual consistency in Bangla summarization, addressing the language's resource scarcity and reducing reliance on reference summaries.
Contribution
It introduces a unified, multilingual instruction-tuned model for question generation, answering, and importance weighting, improving evaluation accuracy and efficiency for Bangla summarization.
Findings
Strong correlation with human judgments (r=0.694, ρ=0.763)
Validated on 300 summaries from educational and medical domains
Provides interpretable diagnostics alongside evaluation scores
Abstract
Evaluating factual consistency is essential for reliable text summarization, particularly in high-stakes domains such as healthcare and news. However, most existing evaluation metrics overlook Bangla, a widely spoken yet under-resourced language, and often depend on reference summaries. We introduce BanglaSummEval, a reference-free, question-answering-based framework for evaluating factual consistency in Bangla summarization. The proposed method assesses both factual accuracy and content coverage through automatically generated questions and answers derived from the source document and the summary. A single multilingual instruction-tuned language model handles question generation, question answering, candidate answer extraction, and question importance weighting. This unified design reduces system complexity and computational cost. To capture semantic consistency beyond surface-level…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsTopic Modeling · Text Readability and Simplification · Biomedical Text Mining and Ontologies
