PARAPHRASUS : A Comprehensive Benchmark for Evaluating Paraphrase Detection Models

TL;DR

PARAPHRASUS is a comprehensive benchmark for evaluating paraphrase detection models across multiple dimensions, revealing trade-offs and enabling tailored assessments for various use cases in NLP.

Contribution

The paper introduces PARAPHRASUS, a multi-dimensional benchmark with diverse datasets for nuanced evaluation of paraphrase detection models, addressing limitations of previous single-metric datasets.

Findings

Models show trade-offs under fine-grained evaluation.

PARAPHRASUS enables calibration for different strictness levels.

Benchmark includes 3 challenges with over 10 datasets.

Abstract

The task of determining whether two texts are paraphrases has long been a challenge in NLP. However, the prevailing notion of paraphrase is often quite simplistic, offering only a limited view of the vast spectrum of paraphrase phenomena. Indeed, we find that evaluating models in a paraphrase dataset can leave uncertainty about their true semantic understanding. To alleviate this, we create PARAPHRASUS, a benchmark designed for multi-dimensional assessment, benchmarking and selection of paraphrase detection models. We find that paraphrase detection models under our fine-grained evaluation lens exhibit trade-offs that cannot be captured through a single classification dataset. Furthermore, PARAPHRASUS allows prompt calibration for different use cases, tailoring LLM models to specific strictness levels. PARAPHRASUS includes 3 challenges spanning over 10 datasets, including 8 repurposed and 2 newly annotated; we release it along with a benchmarking library at https://github.com/impresso/paraphrasus