SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation

TL;DR

This paper introduces SMCLM, a self-supervised causal language model that generates high-quality paraphrases by using semantically meaningful representations, achieving state-of-the-art results in unsupervised paraphrase generation.

Contribution

The paper proposes a novel semantically meaningful causal language modeling approach that enhances autoregressive paraphrase generation without supervision.

Findings

SMCLM produces robust, high-quality paraphrases.

It outperforms existing unsupervised methods.

Current automatic metrics have low reliability for paraphrase evaluation.

Abstract

This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantically meaningful text representation as an initial embedding in the autoregressive training and generation processes. The extensive empirical study demonstrates that the SMCLM approach makes autoregressive models capable of learning robust and high-quality paraphrase generation. The proposed method is competitive with the supervised method and achieves state-of-the-art results in unsupervised approaches. This article also presents a comprehensive set of automatic metrics that cover a wide range of autogenerated paraphrase evaluation aspects. Simultaneously, this article highlights the low reliability of the metrics that are widely used in paraphrase generation evaluation, including BLEU, ROUGE, and BERTScore.