Adapting AlignScore Mertic for Factual Consistency Evaluation of Text in Russian: A Student Abstract

TL;DR

This paper introduces AlignRuScore, an adapted alignment metric for evaluating factual consistency in Russian texts, addressing the lack of suitable tools for non-English NLP applications.

Contribution

The authors adapt the AlignScore metric for Russian by fine-tuning a RuBERT-based model with task-specific heads on Russian and translated datasets.

Findings

Successful adaptation of AlignScore to Russian.

Demonstrated effectiveness of the unified metric for multilingual evaluation.

Released resources to facilitate further research.

Abstract

Ensuring factual consistency in generated text is crucial for reliable natural language processing applications. However, there is a lack of evaluation tools for factual consistency in Russian texts, as existing tools primarily focus on English corpora. To bridge this gap, we introduce AlignRuScore, a comprehensive adaptation of the AlignScore metric for Russian. To adapt the metric, we fine-tuned a RuBERT-based alignment model with task-specific classification and regression heads on Russian and translated English datasets. Our results demonstrate that a unified alignment metric can be successfully ported to Russian, laying the groundwork for robust multilingual factual consistency evaluation. We release the translated corpora, model checkpoints, and code to support further research.