Collocation2Text: Controllable Text Generation from Guide Phrases in Russian

TL;DR

This paper introduces Collocation2Text, a plug-and-play, non-fine-tuning method for controllable Russian text generation using guide phrases, leveraging autoregressive and autoencoding models to produce coherent narratives.

Contribution

The paper presents a novel controllable text generation approach that does not require training, combining ruGPT-3 and ruRoBERTa models to guide narrative development in Russian.

Findings

Effective in generating fluent, coherent texts with user-specified phrases.

Demonstrated success in news article generation tasks.

Improves narrative coherence without additional training.

Abstract

Large pre-trained language models are capable of generating varied and fluent texts. Starting from the prompt, these models generate a narrative that can develop unpredictably. The existing methods of controllable text generation, which guide the narrative in the text in the user-specified direction, require creating a training corpus and an additional time-consuming training procedure. The paper proposes and investigates Collocation2Text, a plug-and-play method for automatic controllable text generation in Russian, which does not require fine-tuning. The method is based on two interacting models: the autoregressive language ruGPT-3 model and the autoencoding language ruRoBERTa model. The idea of the method is to shift the output distribution of the autoregressive model according to the output distribution of the autoencoding model in order to ensure a coherent transition of the narrative in the text towards the guide phrase, which can contain single words or collocations. The autoencoding model, which is able to take into account the left and right contexts of the token, "tells" the autoregressive model which tokens are the most and least logical at the current generation step, increasing or decreasing the probabilities of the corresponding tokens. The experiments on generating news articles using the proposed method showed its effectiveness for automatically generated fluent texts which contain coherent transitions between user-specified phrases.