HeRo: RoBERTa and Longformer Hebrew Language Models

TL;DR

This paper introduces HeRo, a Hebrew language model based on RoBERTa, and LongHeRo, an efficient transformer for long inputs, both achieving state-of-the-art results on multiple NLP tasks for Hebrew.

Contribution

It provides the largest Hebrew pre-training dataset HeDC4 and introduces two models, HeRo and LongHeRo, tailored for standard and long input sequences respectively.

Findings

HeRo and LongHeRo achieved state-of-the-art performance.

The models are evaluated on sentiment analysis, NER, QA, and document classification.

The dataset and models are publicly available.

Abstract

In this paper, we fill in an existing gap in resources available to the Hebrew NLP community by providing it with the largest so far pre-train dataset HeDC4, a state-of-the-art pre-trained language model HeRo for standard length inputs and an efficient transformer LongHeRo for long input sequences. The HeRo model was evaluated on the sentiment analysis, the named entity recognition, and the question answering tasks while the LongHeRo model was evaluated on the document classification task with a dataset composed of long documents. Both HeRo and LongHeRo presented state-of-the-art performance. The dataset and model checkpoints used in this work are publicly available.