Abstractive Summarization of Low resourced Nepali language using Multilingual Transformers

TL;DR

This paper investigates the use of multilingual transformer models, specifically mBART and mT5, for abstractive summarization of Nepali news articles, addressing the low-resource challenge and evaluating with ROUGE scores and human judgment.

Contribution

It introduces a new Nepali news summarization dataset and demonstrates the effectiveness of 4-bit quantized mBART with LoRA for generating high-quality summaries.

Findings

4-bit quantized mBART with LoRA outperforms other models

Model selected 34.05% of the time in human evaluation

Fine-tuned models achieve better ROUGE scores

Abstract

Automatic text summarization in Nepali language is an unexplored area in natural language processing (NLP). Although considerable research has been dedicated to extractive summarization, the area of abstractive summarization, especially for low-resource languages such as Nepali, remains largely unexplored. This study explores the use of multilingual transformer models, specifically mBART and mT5, for generating headlines for Nepali news articles through abstractive summarization. The research addresses key challenges associated with summarizing texts in Nepali by first creating a summarization dataset through web scraping from various Nepali news portals. These multilingual models were then fine-tuned using different strategies. The performance of the fine-tuned models were then assessed using ROUGE scores and human evaluation to ensure the generated summaries were coherent and conveyed the original meaning. During the human evaluation, the participants were asked to select the best summary among those generated by the models, based on criteria such as relevance, fluency, conciseness, informativeness, factual accuracy, and coverage. During the evaluation with ROUGE scores, the 4-bit quantized mBART with LoRA model was found to be effective in generating better Nepali news headlines in comparison to other models and also it was selected 34.05% of the time during the human evaluation, outperforming all other fine-tuned models created for Nepali News headline generation.