Fine-tuning GPT-3 for Russian Text Summarization

TL;DR

This paper demonstrates how fine-tuning GPT-3 for Russian text summarization can outperform existing models, despite some issues with factual accuracy and entity preservation.

Contribution

It shows that fine-tuning ruGPT-3 on Russian news data with hyperparameter tuning improves summarization performance without changing the model architecture.

Findings

Outperforms state-of-the-art models on Russian summarization tasks

Hyperparameter tuning reduces randomness and improves relevance

Model still struggles with entity accuracy and factual consistency

Abstract

Automatic summarization techniques aim to shorten and generalize information given in the text while preserving its core message and the most relevant ideas. This task can be approached and treated with a variety of methods, however, not many attempts have been made to produce solutions specifically for the Russian language despite existing localizations of the state-of-the-art models. In this paper, we aim to showcase ruGPT3 ability to summarize texts, fine-tuning it on the corpora of Russian news with their corresponding human-generated summaries. Additionally, we employ hyperparameter tuning so that the model's output becomes less random and more tied to the original text. We evaluate the resulting texts with a set of metrics, showing that our solution can surpass the state-of-the-art model's performance without additional changes in architecture or loss function. Despite being able to produce sensible summaries, our model still suffers from a number of flaws, namely, it is prone to altering Named Entities present in the original text (such as surnames, places, dates), deviating from facts stated in the given document, and repeating the information in the summary.