Towards Human Understanding of Paraphrase Types in Large Language Models

TL;DR

This paper introduces a detailed dataset and analysis of how large language models, like ChatGPT, perform on various linguistic paraphrase types, revealing strengths and limitations in their understanding of complex language structures.

Contribution

The study presents APTY, a new dataset of 800 annotated paraphrases with human preferences, and evaluates LLMs' abilities to handle diverse paraphrase types, highlighting areas for improvement.

Findings

ChatGPT excels at simple paraphrase types like additions and deletions.

Models struggle with complex structures such as subordination changes.

The dataset can be used to fine-tune models for specific linguistic capabilities.

Abstract

Paraphrases represent a human's intuitive ability to understand expressions presented in various different ways. Current paraphrase evaluations of language models primarily use binary approaches, offering limited interpretability of specific text changes. Atomic paraphrase types (APT) decompose paraphrases into different linguistic changes and offer a granular view of the flexibility in linguistic expression (e.g., a shift in syntax or vocabulary used). In this study, we assess the human preferences towards ChatGPT in generating English paraphrases with ten APTs and five prompting techniques. We introduce APTY (Atomic Paraphrase TYpes), a dataset of 800 sentence-level and word-level annotations by 15 annotators. The dataset also provides a human preference ranking of paraphrases with different types that can be used to fine-tune models with RLHF and DPO methods. Our results reveal that ChatGPT and a DPO-trained LLama 7B model can generate simple APTs, such as additions and deletions, but struggle with complex structures (e.g., subordination changes). This study contributes to understanding which aspects of paraphrasing language models have already succeeded at understanding and what remains elusive. In addition, we show how our curated datasets can be used to develop language models with specific linguistic capabilities.