HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew

TL;DR

HeSum is a new Hebrew dataset with 10,000 article-summary pairs, designed to evaluate and advance abstractive summarization in Hebrew, addressing linguistic challenges and resource gaps for low-resourced languages.

Contribution

The paper introduces HeSum, a large, professionally curated Hebrew summarization dataset, and analyzes its linguistic complexity and challenges for current language models.

Findings

HeSum reveals significant difficulties for state-of-the-art LLMs in Hebrew summarization.

HeSum's linguistic features confirm high abstractness and morphological complexity.

The dataset serves as a benchmark for future research in Hebrew NLP.

Abstract

While large language models (LLMs) excel in various natural language tasks in English, their performance in lower-resourced languages like Hebrew, especially for generative tasks such as abstractive summarization, remains unclear. The high morphological richness in Hebrew adds further challenges due to the ambiguity in sentence comprehension and the complexities in meaning construction. In this paper, we address this resource and evaluation gap by introducing HeSum, a novel benchmark specifically designed for abstractive text summarization in Modern Hebrew. HeSum consists of 10,000 article-summary pairs sourced from Hebrew news websites written by professionals. Linguistic analysis confirms HeSum's high abstractness and unique morphological challenges. We show that HeSum presents distinct difficulties for contemporary state-of-the-art LLMs, establishing it as a valuable testbed for generative language technology in Hebrew, and MRLs generative challenges in general.