Semi-Supervised Exaggeration Detection of Health Science Press Releases

TL;DR

This paper formalizes the problem of exaggeration detection in science communication, creates a benchmark dataset, and introduces MT-PET, a multi-task learning method that improves detection performance especially in low-data scenarios.

Contribution

The paper introduces a new benchmark dataset for exaggeration detection in press releases and proposes MT-PET, a novel multi-task learning approach that enhances few-shot learning capabilities.

Findings

MT-PET outperforms PET and supervised learning in limited data settings.

The dataset enables benchmarking of machine learning models on exaggeration detection.

MT-PET leverages knowledge from QA tasks to improve performance.

Abstract

Public trust in science depends on honest and factual communication of scientific papers. However, recent studies have demonstrated a tendency of news media to misrepresent scientific papers by exaggerating their findings. Given this, we present a formalization of and study into the problem of exaggeration detection in science communication. While there are an abundance of scientific papers and popular media articles written about them, very rarely do the articles include a direct link to the original paper, making data collection challenging. We address this by curating a set of labeled press release/abstract pairs from existing expert annotated studies on exaggeration in press releases of scientific papers suitable for benchmarking the performance of machine learning models on the task. Using limited data from this and previous studies on exaggeration detection in science, we introduce MT-PET, a multi-task version of Pattern Exploiting Training (PET), which leverages knowledge from complementary cloze-style QA tasks to improve few-shot learning. We demonstrate that MT-PET outperforms PET and supervised learning both when data is limited, as well as when there is an abundance of data for the main task.